Oct 30 16:17:58.278953 ip-10-0-128-181 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 16:17:58.278963 ip-10-0-128-181 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 16:17:58.278970 ip-10-0-128-181 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 16:17:58.279189 ip-10-0-128-181 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 16:18:08.295166 ip-10-0-128-181 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 16:18:08.295188 ip-10-0-128-181 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 32273130ad91492e8ece9ba5f5249a7d -- Oct 30 16:20:17.906864 ip-10-0-128-181 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:20:18.314189 ip-10-0-128-181 kubenswrapper[2579]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:18.314189 ip-10-0-128-181 kubenswrapper[2579]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:20:18.314189 ip-10-0-128-181 kubenswrapper[2579]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:18.314189 ip-10-0-128-181 kubenswrapper[2579]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:20:18.319149 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319070 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:18.319149 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319148 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319154 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319157 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319159 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319162 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319165 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319168 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319170 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319173 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319176 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319179 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319181 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319184 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319186 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319190 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319193 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319196 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319198 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319201 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:18.319220 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319204 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319207 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319209 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319212 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319214 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319217 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319219 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319221 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319224 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319227 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319230 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319232 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319235 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319238 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319240 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319242 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319245 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319247 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319249 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319252 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:18.319707 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319254 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319257 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319259 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319262 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319264 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319266 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319269 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319271 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319274 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319277 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319279 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319282 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319284 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319287 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319290 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319293 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319295 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319298 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319300 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:18.320455 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319304 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319306 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319309 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319311 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319313 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319316 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319320 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319323 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319327 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319329 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319332 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319334 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319337 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319339 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319343 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319347 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319350 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319352 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319355 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:18.321270 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319358 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319360 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319363 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319366 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319369 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319371 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319374 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319377 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319380 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319383 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319385 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319388 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319390 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319393 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319396 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319398 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319401 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319403 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319405 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319408 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:18.322090 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319412 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319414 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319417 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319419 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319421 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319424 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319426 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.319429 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321930 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321949 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321955 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321960 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321964 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321969 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321974 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321978 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321982 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321987 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321991 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:18.322956 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321995 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.321999 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322004 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322008 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322011 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322016 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322020 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322025 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322029 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322034 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322038 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322042 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322046 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322050 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322055 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322059 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322063 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322067 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322072 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322076 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:18.323794 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322081 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322085 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322089 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322092 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322096 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322100 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322104 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322107 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322111 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322115 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322120 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322124 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322129 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322133 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322137 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322141 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322145 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322150 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322154 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:18.324322 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322158 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322162 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322166 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322171 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322175 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322189 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322194 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322199 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322203 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322207 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322211 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322216 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322221 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322226 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322230 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322234 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322238 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322242 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322246 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322250 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:18.324825 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322254 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322258 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322263 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322268 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322273 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322277 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322283 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322289 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322294 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322298 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322302 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322307 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322311 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322317 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322321 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322325 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322329 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322333 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322337 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:18.325388 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322342 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322346 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322350 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322354 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322358 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322363 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322367 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322371 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322375 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322379 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322383 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322388 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322392 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322396 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322402 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322410 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.322415 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323264 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323276 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:18.326243 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323281 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323286 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323290 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323294 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323299 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323303 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323307 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323312 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323317 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323321 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323325 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323329 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323333 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323338 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323343 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323347 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323351 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323355 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323359 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:18.327075 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323363 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323368 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323372 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323376 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323380 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323384 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323388 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323392 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323397 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323401 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323405 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323410 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323414 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323418 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323422 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323426 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323430 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323435 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323439 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323443 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:18.327772 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323448 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323452 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323456 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323461 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323464 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323469 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323473 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323478 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323482 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323486 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323490 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323494 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323498 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323502 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323507 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323511 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323515 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323519 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323523 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323527 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:18.328382 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323531 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323535 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323564 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323569 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323574 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323578 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323585 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323592 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323596 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323601 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323605 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323610 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323615 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323619 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323625 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323629 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323633 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323637 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323641 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:18.329020 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323645 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323652 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323658 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323664 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323668 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323674 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323678 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323683 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323687 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323691 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323695 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323700 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323703 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323708 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323712 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323716 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323721 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323725 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323729 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:18.329462 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323733 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323739 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323743 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323747 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323751 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323755 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.323759 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325272 2579 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325296 2579 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325307 2579 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325315 2579 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325322 2579 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325327 2579 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325335 2579 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325341 2579 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325346 2579 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325351 2579 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325357 2579 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325363 2579 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325368 2579 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325372 2579 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325377 2579 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325381 2579 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:20:18.330023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325385 2579 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325390 2579 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325397 2579 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325401 2579 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325406 2579 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325410 2579 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325416 2579 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325422 2579 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325427 2579 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325433 2579 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325437 2579 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325442 2579 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325447 2579 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325452 2579 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325457 2579 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325461 2579 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325468 2579 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325473 2579 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325478 2579 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325483 2579 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325489 2579 flags.go:64] FLAG: --enable-server="true" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325493 2579 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325500 2579 flags.go:64] FLAG: --event-burst="100" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325505 2579 flags.go:64] FLAG: --event-qps="50" Oct 30 16:20:18.330569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325509 2579 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325514 2579 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325519 2579 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325526 2579 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325531 2579 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325536 2579 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325556 2579 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325561 2579 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325566 2579 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325570 2579 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325575 2579 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325579 2579 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325584 2579 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325588 2579 flags.go:64] FLAG: --feature-gates="" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325598 2579 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325603 2579 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325608 2579 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325613 2579 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325618 2579 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325623 2579 flags.go:64] FLAG: --help="false" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325628 2579 flags.go:64] FLAG: --hostname-override="ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325633 2579 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325638 2579 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:20:18.331181 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325643 2579 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325648 2579 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325653 2579 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325658 2579 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325663 2579 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325669 2579 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325674 2579 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325679 2579 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325684 2579 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325689 2579 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325693 2579 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325697 2579 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325703 2579 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325707 2579 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325712 2579 flags.go:64] FLAG: --lock-file="" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325716 2579 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325722 2579 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325727 2579 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325743 2579 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325748 2579 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325753 2579 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325758 2579 flags.go:64] FLAG: --logging-format="text" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325762 2579 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325770 2579 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:20:18.331776 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325774 2579 flags.go:64] FLAG: --manifest-url="" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325779 2579 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325786 2579 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325791 2579 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325806 2579 flags.go:64] FLAG: --max-pods="110" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325811 2579 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325815 2579 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325820 2579 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325825 2579 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325829 2579 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325834 2579 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325839 2579 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325851 2579 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325855 2579 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325861 2579 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325867 2579 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325871 2579 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325875 2579 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325881 2579 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325885 2579 flags.go:64] FLAG: --port="10250" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325890 2579 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325895 2579 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-051aebc9d776fd5d0" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325900 2579 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325905 2579 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325909 2579 flags.go:64] FLAG: --register-node="true" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325918 2579 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:20:18.332332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325925 2579 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325930 2579 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325934 2579 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325939 2579 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325945 2579 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325951 2579 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325956 2579 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325961 2579 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325966 2579 flags.go:64] FLAG: --runonce="false" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325971 2579 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325976 2579 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325981 2579 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325985 2579 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325990 2579 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325995 2579 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.325999 2579 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326004 2579 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326009 2579 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326013 2579 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326018 2579 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326023 2579 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326029 2579 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326034 2579 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326039 2579 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326047 2579 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:20:18.333058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326052 2579 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326056 2579 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326063 2579 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326068 2579 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326072 2579 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326077 2579 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326083 2579 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326087 2579 flags.go:64] FLAG: --v="2" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326094 2579 flags.go:64] FLAG: --version="false" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326101 2579 flags.go:64] FLAG: --vmodule="" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326107 2579 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326112 2579 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326323 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326330 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326336 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326340 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326344 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326349 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326353 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326357 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326362 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:18.333678 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326369 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326375 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326381 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326386 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326390 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326394 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326399 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326404 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326409 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326413 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326417 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326422 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326426 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326431 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326435 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326439 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326443 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326449 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326453 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326458 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:18.334194 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326462 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326466 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326470 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326475 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326478 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326483 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326487 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326492 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326496 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326501 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326505 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326509 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326513 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326517 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326521 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326525 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326530 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326535 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326559 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326563 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:18.334683 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326567 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326573 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326577 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326581 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326585 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326589 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326593 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326596 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326600 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326609 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326615 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326621 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326626 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326630 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326634 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326638 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326643 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326647 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326652 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:18.335153 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326656 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326660 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326664 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326668 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326673 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326677 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326681 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326685 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326690 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326694 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326698 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326702 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326706 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326710 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326715 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326720 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326725 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326729 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326733 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326737 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:18.335687 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326742 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326746 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326751 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326756 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326760 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326765 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326769 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326774 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326778 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326782 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326786 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326790 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326794 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326798 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326803 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326807 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326811 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:18.336161 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326815 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.326857 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326921 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326929 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326934 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326938 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326944 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326948 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326952 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326956 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326960 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326967 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326972 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326976 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326980 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326984 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:18.336671 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326988 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326993 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.326998 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327002 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327007 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327011 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327015 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327019 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327023 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327027 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327031 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327036 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327040 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327044 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327049 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327053 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327059 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327063 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327068 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327072 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:18.337046 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327076 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327080 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327084 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327088 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327092 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327096 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327100 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327104 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327109 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327114 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327118 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327123 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327127 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327131 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327136 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327140 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327144 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327149 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327156 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:18.337516 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327162 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327167 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327172 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327177 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327181 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327185 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327189 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327193 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327198 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327203 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327208 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327212 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327217 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327221 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327225 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327229 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327233 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327237 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327241 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327245 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:18.338852 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327249 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327254 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327263 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327267 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327271 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327275 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327279 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327284 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327289 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327293 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327297 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327302 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327306 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327310 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327314 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327318 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327323 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327327 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327331 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:18.339317 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327335 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327339 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327345 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327350 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327354 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327358 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327362 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327367 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327371 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327380 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327385 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327390 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327394 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.327399 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.334579 2579 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.334595 2579 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.335427 2579 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.335434 2579 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335493 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:18.339768 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335500 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335503 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335506 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335509 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335512 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335515 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335518 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335520 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335523 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335525 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335528 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335531 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335534 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335537 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335558 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335561 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335564 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335566 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335569 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:18.340219 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335571 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335574 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335576 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335579 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335582 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335584 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335587 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335590 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335592 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335596 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335599 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335602 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335605 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335607 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335610 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335613 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335615 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335618 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335621 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:18.340685 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335623 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335626 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335628 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335631 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335633 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335636 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335638 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335641 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335643 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335646 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335648 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335651 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335654 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335657 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335659 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335661 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335664 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335666 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335669 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335672 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:18.341131 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335674 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335677 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335680 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335682 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335685 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335688 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335691 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335693 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335696 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335699 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335701 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335705 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335707 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335710 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335712 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335715 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335717 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335719 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335722 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335724 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:18.341651 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335727 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335729 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335731 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335734 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335736 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335739 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335741 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335744 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335746 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335749 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335751 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335754 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335756 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335759 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335761 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335764 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335767 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335769 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335772 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:18.342123 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335775 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335777 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335780 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335783 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335787 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335790 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335792 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335795 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.335822 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335957 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335962 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335965 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335968 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335971 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335974 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335976 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:18.342578 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335979 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335981 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335984 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335987 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335989 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335992 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335994 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335996 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.335999 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336001 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336004 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336007 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336010 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336013 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336015 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336019 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336021 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336025 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336028 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:18.342963 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336031 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336034 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336036 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336038 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336041 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336043 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336046 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336048 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336051 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336053 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336055 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336058 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336060 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336063 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336065 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336068 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336070 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336073 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336075 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336078 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:18.343402 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336081 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336083 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336086 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336088 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336091 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336093 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336096 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336098 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336100 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336103 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336107 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336111 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336114 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336116 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336119 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336121 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336124 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336126 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336129 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:18.343896 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336131 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336134 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336136 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336139 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336141 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336144 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336146 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336148 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336151 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336153 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336156 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336158 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336160 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336163 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336166 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336169 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336171 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336174 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336176 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336178 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:18.344342 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336181 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336183 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336186 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336189 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336192 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336194 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336196 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336199 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336201 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336204 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336206 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336208 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336211 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336214 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336216 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336218 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336221 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336223 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336226 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:18.344910 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336228 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336231 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.336253 2579 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336286 2579 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336290 2579 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336294 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336297 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336299 2579 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336302 2579 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336304 2579 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336307 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336309 2579 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336312 2579 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336314 2579 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336316 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336319 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:20:18.346238 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336321 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336324 2579 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336327 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336329 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336332 2579 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336334 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336336 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336339 2579 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336341 2579 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336343 2579 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336346 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336348 2579 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336351 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336353 2579 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336356 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336358 2579 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336361 2579 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336363 2579 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336366 2579 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:20:18.346625 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336368 2579 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336371 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336373 2579 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336375 2579 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336378 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336380 2579 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336382 2579 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336385 2579 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336388 2579 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336390 2579 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336393 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336395 2579 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336398 2579 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336400 2579 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336403 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336405 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336408 2579 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336412 2579 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336415 2579 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:20:18.347064 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336418 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336421 2579 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336424 2579 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336426 2579 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336429 2579 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336431 2579 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336435 2579 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336439 2579 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336442 2579 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336444 2579 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336447 2579 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336449 2579 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336452 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336454 2579 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336457 2579 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336459 2579 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336462 2579 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336464 2579 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336467 2579 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336469 2579 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:20:18.347496 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336472 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336474 2579 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336477 2579 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336479 2579 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336482 2579 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336484 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336487 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336489 2579 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336492 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336499 2579 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336503 2579 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336506 2579 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336508 2579 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336511 2579 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336514 2579 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336516 2579 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336519 2579 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336521 2579 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336525 2579 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:20:18.347983 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336529 2579 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336532 2579 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336534 2579 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336553 2579 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336556 2579 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336559 2579 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336561 2579 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336564 2579 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336567 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336569 2579 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336572 2579 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336574 2579 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336577 2579 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336581 2579 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336583 2579 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:18.336586 2579 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.336633 2579 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.338635 2579 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 16:20:18.348418 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.339458 2579 server.go:1014] "Starting client certificate rotation" Oct 30 16:20:18.348853 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.339571 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:20:18.348853 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.339604 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:20:18.348853 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.341767 2579 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:20:18.348853 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.346404 2579 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:20:18.356434 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.356416 2579 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:20:18.359862 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.359846 2579 log.go:25] "Validated CRI v1 image API" Oct 30 16:20:18.360653 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.360626 2579 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:20:18.365176 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.365152 2579 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a2017790-fd04-4ac0-81c7-2185d1e2f434:/dev/nvme0n1p3 dbdda714-2453-4b06-b787-c206bcd7dfe0:/dev/nvme0n1p4] Oct 30 16:20:18.365236 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.365176 2579 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 16:20:18.372267 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.372162 2579 manager.go:217] Machine: {Timestamp:2026-10-30 16:20:18.369421709 +0000 UTC m=+0.343863724 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3084848 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e7bb25d1b648c472416a203ec6f1a SystemUUID:ec2e7bb2-5d1b-648c-4724-16a203ec6f1a BootID:32273130-ad91-492e-8ece-9ba5f5249a7d Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:dd:52:16:bd Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:dd:52:16:bd Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6a:2e:a1:99:ec:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:20:18.372267 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.372265 2579 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:20:18.372390 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.372375 2579 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:20:18.374999 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.374975 2579 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:20:18.375135 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.375001 2579 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-181.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:20:18.375177 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.375138 2579 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:20:18.375177 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.375146 2579 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:20:18.375177 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.375164 2579 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:20:18.375251 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.375180 2579 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:20:18.376027 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.376017 2579 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:20:18.380208 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.380197 2579 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:20:18.380330 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.380321 2579 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:20:18.381158 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.381148 2579 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:20:18.381195 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.381165 2579 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:20:18.381195 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.381183 2579 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:20:18.381243 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.381196 2579 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:20:18.381243 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.381208 2579 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:20:18.381627 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.381615 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:20:18.383835 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.383817 2579 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:20:18.389091 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.389078 2579 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:20:18.389974 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.389962 2579 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:20:18.390019 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.389992 2579 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:20:18.390138 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390128 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:20:18.390169 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390144 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:20:18.390169 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390157 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:20:18.390169 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390164 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:20:18.390169 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390170 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:20:18.390268 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390175 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:20:18.390787 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390778 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:20:18.390816 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390788 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:20:18.390816 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390795 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:20:18.390816 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390801 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:20:18.390816 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390809 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:20:18.391009 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.390818 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:20:18.391643 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.391634 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:20:18.391679 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.391644 2579 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:20:18.395388 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.395376 2579 server.go:1263] "Started kubelet" Oct 30 16:20:18.395467 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.395443 2579 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:20:18.395578 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.395498 2579 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:20:18.395629 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.395606 2579 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:20:18.396052 ip-10-0-128-181 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:20:18.396786 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.396769 2579 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:20:18.398267 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.398251 2579 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:20:18.403105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.403088 2579 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:20:18.403447 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.403426 2579 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 16:20:18.403618 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.403600 2579 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:20:18.403709 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.403685 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:18.404442 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.404306 2579 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:20:18.404442 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.404445 2579 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:20:18.404972 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.404895 2579 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:20:18.404972 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.404912 2579 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:20:18.405298 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.405158 2579 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:20:18.405298 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.405236 2579 factory.go:55] Registering systemd factory Oct 30 16:20:18.405298 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.405253 2579 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:20:18.405533 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.405519 2579 factory.go:153] Registering CRI-O factory Oct 30 16:20:18.405612 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.405559 2579 factory.go:223] Registration of the crio container factory successfully Oct 30 16:20:18.405612 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.405606 2579 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:20:18.405708 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.405625 2579 factory.go:103] Registering Raw factory Oct 30 16:20:18.405708 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.405639 2579 manager.go:1199] Started watching for new ooms in manager Oct 30 16:20:18.406799 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.406783 2579 manager.go:319] Starting recovery of all containers Oct 30 16:20:18.417232 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.417215 2579 manager.go:324] Recovery completed Oct 30 16:20:18.421704 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.421521 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:18.423398 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.423380 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:18.423452 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.423415 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:18.423452 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.423425 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:18.423936 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.423922 2579 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:20:18.423936 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.423935 2579 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:20:18.424092 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.423946 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:18.424092 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.423954 2579 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:20:18.425017 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.425001 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:18.425098 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.425030 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:18.425098 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.425045 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:18.426053 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.426041 2579 policy_none.go:50] "Start" Oct 30 16:20:18.426100 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.426056 2579 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:20:18.426100 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.426065 2579 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:20:18.428030 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.428019 2579 policy_none.go:44] "Start" Oct 30 16:20:18.429803 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.429772 2579 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-128-181.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 16:20:18.429882 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.429778 2579 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-128-181.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 16:20:18.431914 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.431897 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:20:18.431979 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.431946 2579 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kpfr8" Oct 30 16:20:18.442331 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.441107 2579 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-128-181.ec2.internal.18e35b1412bba36a default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-128-181.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-128-181.ec2.internal,},FirstTimestamp:2026-10-30 16:20:18.395349866 +0000 UTC m=+0.369791878,LastTimestamp:2026-10-30 16:20:18.395349866 +0000 UTC m=+0.369791878,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-128-181.ec2.internal,}" Oct 30 16:20:18.443650 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.443631 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-kpfr8" Oct 30 16:20:18.450581 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.450538 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-128-181.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 16:20:18.450667 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.450640 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 16:20:18.450806 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.450775 2579 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 16:20:18.469316 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.469184 2579 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:20:18.473568 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.473538 2579 manager.go:341] "Starting Device Plugin manager" Oct 30 16:20:18.473642 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.473592 2579 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:20:18.473642 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.473601 2579 server.go:87] "Starting device plugin registration server" Oct 30 16:20:18.473808 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.473789 2579 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:20:18.473860 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.473808 2579 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:20:18.475722 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.474287 2579 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:20:18.475722 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.474366 2579 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:20:18.475722 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.474374 2579 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:20:18.476081 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.476055 2579 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:20:18.476175 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.476108 2579 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:18.561898 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.561870 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:20:18.563597 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.563259 2579 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:20:18.563760 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.563750 2579 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:20:18.563799 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.563778 2579 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:20:18.563837 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.563807 2579 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 16:20:18.574399 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.574354 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:18.575076 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.575061 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:18.575141 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.575088 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:18.575141 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.575098 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:18.575141 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.575119 2579 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.579783 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.579770 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:18.596066 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.596047 2579 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.596137 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.596066 2579 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-128-181.ec2.internal\": node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:18.639399 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.639383 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:18.664861 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.664839 2579 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal"] Oct 30 16:20:18.664954 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.664893 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:18.665656 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.665640 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:18.665740 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.665675 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:18.665740 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.665693 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:18.667797 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.667781 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:18.667935 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.667922 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.667996 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.667951 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:18.668433 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.668415 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:18.668527 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.668440 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:18.668527 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.668451 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:18.668527 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.668415 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:18.668698 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.668527 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:18.668698 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.668560 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:18.670468 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.670452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.670566 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.670479 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:18.671216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.671198 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:18.671294 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.671226 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:18.671294 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.671235 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:18.700121 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.700100 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-181.ec2.internal\" not found" node="ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.703040 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.703024 2579 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-128-181.ec2.internal\" not found" node="ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.739803 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.739785 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:18.806378 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.806346 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.806378 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.806382 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.806532 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.806403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c2020069c5384399725929d6ee8319e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-181.ec2.internal\" (UID: \"3c2020069c5384399725929d6ee8319e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.840306 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.840254 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:18.906750 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.906724 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.906862 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.906693 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.906862 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.906807 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.906862 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.906835 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c2020069c5384399725929d6ee8319e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-181.ec2.internal\" (UID: \"3c2020069c5384399725929d6ee8319e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.906976 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.906869 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c2020069c5384399725929d6ee8319e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-181.ec2.internal\" (UID: \"3c2020069c5384399725929d6ee8319e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.906976 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:18.906884 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:20:18.940677 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:18.940657 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:19.004102 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.004077 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:20:19.006240 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.006225 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:20:19.041117 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:19.041094 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:19.141669 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:19.141590 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:19.242128 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:19.242099 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:19.339685 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.339660 2579 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 16:20:19.340113 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.339793 2579 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 16:20:19.342821 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:19.342801 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:19.443911 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:19.443855 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:19.448321 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.448291 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:15:19 +0000 UTC" deadline="2028-03-26 18:16:24.467155273 +0000 UTC" Oct 30 16:20:19.448321 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.448318 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12313h56m5.018840436s" Oct 30 16:20:19.477852 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.477832 2579 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:19.481073 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:19.481042 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c2020069c5384399725929d6ee8319e.slice/crio-6d0ffdfc11a08f5cc3957740a7e3ef60833c74db082289fa4e381f10835cc270 WatchSource:0}: Error finding container 6d0ffdfc11a08f5cc3957740a7e3ef60833c74db082289fa4e381f10835cc270: Status 404 returned error can't find the container with id 6d0ffdfc11a08f5cc3957740a7e3ef60833c74db082289fa4e381f10835cc270 Oct 30 16:20:19.488135 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.488122 2579 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:20:19.493614 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:19.493591 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc459e3b80b3dfd1e7e8df0c149b8ceb5.slice/crio-a42f88609e94dc9ee337ffff0f37772fbaef9f59f4cdf4df369a135c2a1bba4c WatchSource:0}: Error finding container a42f88609e94dc9ee337ffff0f37772fbaef9f59f4cdf4df369a135c2a1bba4c: Status 404 returned error can't find the container with id a42f88609e94dc9ee337ffff0f37772fbaef9f59f4cdf4df369a135c2a1bba4c Oct 30 16:20:19.522091 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.522074 2579 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:20:19.522907 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.522891 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:20:19.522988 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.522924 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:20:19.522988 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.522938 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:20:19.544226 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:19.544207 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:19.566926 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.566891 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" event={"ID":"c459e3b80b3dfd1e7e8df0c149b8ceb5","Type":"ContainerStarted","Data":"a42f88609e94dc9ee337ffff0f37772fbaef9f59f4cdf4df369a135c2a1bba4c"} Oct 30 16:20:19.567866 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.567844 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" event={"ID":"3c2020069c5384399725929d6ee8319e","Type":"ContainerStarted","Data":"6d0ffdfc11a08f5cc3957740a7e3ef60833c74db082289fa4e381f10835cc270"} Oct 30 16:20:19.644403 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:19.644369 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:19.668888 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.668863 2579 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:19.745397 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:19.745336 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:19.846084 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:19.846042 2579 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:20:19.891705 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.891676 2579 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:19.903963 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.903906 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:20:19.929714 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.929693 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:19.930894 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.930873 2579 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:20:19.953588 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:19.953490 2579 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:20:21.409377 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.409348 2579 apiserver.go:52] "Watching apiserver" Oct 30 16:20:21.431749 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.431726 2579 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 16:20:21.440654 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.440626 2579 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:20:21.440992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.440970 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal","openshift-multus/multus-2bm4t","openshift-multus/multus-additional-cni-plugins-n8mr5","kube-system/global-pull-secret-syncer-26nhz","openshift-network-diagnostics/network-check-target-gccnz","kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal","openshift-cluster-node-tuning-operator/tuned-772f2","openshift-dns/node-resolver-68psh","openshift-multus/network-metrics-daemon-9ls8t","openshift-network-operator/iptables-alerter-p528z","openshift-ovn-kubernetes/ovnkube-node-w6x6l","kube-system/konnectivity-agent-glrfx","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj","openshift-image-registry/node-ca-rns4g"] Oct 30 16:20:21.443729 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.443709 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.446017 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.445964 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.448418 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.448392 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.451091 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.450890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:21.451091 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.450963 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:21.453288 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.453108 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:21.453288 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.453183 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:21.455442 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.455422 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.457644 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.457628 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:21.457736 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.457687 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:21.459936 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.459875 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.460589 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.460572 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gfrzf\"" Oct 30 16:20:21.460589 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.460576 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:21.460779 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.460764 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:21.463472 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.463455 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.466435 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.466095 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.468834 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.468816 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.470963 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.470945 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:21.491053 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.491020 2579 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 16:20:21.491186 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.491093 2579 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t4dtb" Oct 30 16:20:21.534195 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.534180 2579 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:20:21.543429 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543403 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-run\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.543523 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543464 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-tmp\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.543523 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.543639 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb25da1d-2305-4f75-9920-e52fe596f5b7-hosts-file\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.543639 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543631 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvtgq\" (UniqueName: \"kubernetes.io/projected/cb25da1d-2305-4f75-9920-e52fe596f5b7-kube-api-access-cvtgq\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.543735 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.543735 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543703 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12134b5c-de72-4972-a185-3d5450c7b5e7-agent-certs\") pod \"konnectivity-agent-glrfx\" (UID: \"12134b5c-de72-4972-a185-3d5450c7b5e7\") " pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:21.543821 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543732 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72976414-f225-493b-9c55-3ab89bb831a8-serviceca\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.543821 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543755 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:21.543894 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-script-lib\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.543894 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-system-cni-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.543894 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543853 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:21.543894 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543877 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-bin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.544008 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543905 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-lib-modules\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.544008 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwsxf\" (UniqueName: \"kubernetes.io/projected/72976414-f225-493b-9c55-3ab89bb831a8-kube-api-access-bwsxf\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.544008 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543946 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-netns\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.544008 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.543996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-netd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.544136 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544011 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.544136 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544024 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12134b5c-de72-4972-a185-3d5450c7b5e7-konnectivity-ca\") pod \"konnectivity-agent-glrfx\" (UID: \"12134b5c-de72-4972-a185-3d5450c7b5e7\") " pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:21.544136 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-etc-tuned\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.544136 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544057 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-system-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.544136 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544071 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-var-lib-kubelet\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.544136 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysconfig\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.544136 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544106 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-sys\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.544136 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544120 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e50ac2ed-9597-47ec-a4be-078d412f0b61-iptables-alerter-script\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-dbus\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544181 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-etc-kubernetes\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544194 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e50ac2ed-9597-47ec-a4be-078d412f0b61-host-slash\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544208 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-cni-binary-copy\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544272 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-conf-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544298 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-multus-certs\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25da1d-2305-4f75-9920-e52fe596f5b7-tmp-dir\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544335 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-k8s-cni-cncf-io\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544355 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-kubelet\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.544370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544368 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-registration-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544388 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-device-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544406 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-kubelet-config\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544419 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-socket-dir-parent\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544432 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-conf\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544445 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-slash\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544474 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-ovn\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544486 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-node-log\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544499 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ds4gb\" (UniqueName: \"kubernetes.io/projected/be63e086-3851-459b-9123-262570be3b86-kube-api-access-ds4gb\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544512 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qkbpx\" (UniqueName: \"kubernetes.io/projected/cc9755b7-6248-49a7-93c2-06a74060d379-kube-api-access-qkbpx\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544531 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-kubelet\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544543 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-modprobe-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544558 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-kubernetes\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544581 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgv82\" (UniqueName: \"kubernetes.io/projected/e50ac2ed-9597-47ec-a4be-078d412f0b61-kube-api-access-sgv82\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544596 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.544676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544637 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544671 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw7vq\" (UniqueName: \"kubernetes.io/projected/5874f474-b442-4eec-a808-33997e342cb5-kube-api-access-lw7vq\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544684 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-systemd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544703 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-bin\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544819 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cnibin\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544833 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqjj\" (UniqueName: \"kubernetes.io/projected/45b113ed-a156-4c53-bee5-26707e5717c3-kube-api-access-5hqjj\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544861 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-socket-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-hostroot\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544919 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-systemd-units\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544940 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-log-socket\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544956 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-config\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544969 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.544982 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-cnibin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545000 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-multus\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545014 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72976414-f225-493b-9c55-3ab89bb831a8-host\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545031 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x66t\" (UniqueName: \"kubernetes.io/projected/569992a0-aa75-48cb-ad8d-fb61295a5093-kube-api-access-4x66t\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:21.545105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545044 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-var-lib-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545069 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/be63e086-3851-459b-9123-262570be3b86-ovn-node-metrics-cert\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545082 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn2zm\" (UniqueName: \"kubernetes.io/projected/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-kube-api-access-vn2zm\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545100 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-netns\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545125 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-multus-daemon-config\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545138 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-systemd\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-host\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545167 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-etc-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545188 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545201 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-env-overrides\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545214 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545234 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545257 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-sys-fs\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.545539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.545270 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.561626 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.561481 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:20:21.561626 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.561571 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:20:21.561866 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.561718 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:20:21.561866 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.561820 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n7jsz\"" Oct 30 16:20:21.561988 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.561865 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:20:21.561988 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.561931 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:20:21.561988 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.561949 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:20:21.563462 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.563300 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:20:21.563462 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.563351 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:20:21.563462 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.563459 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:20:21.563681 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.563487 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:20:21.563681 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.563527 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:20:21.563681 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.563528 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.563871 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5kvnn\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.563910 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564001 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sg8tz\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564108 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564148 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564185 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564248 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564265 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564317 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564346 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l28xz\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564356 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564450 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564524 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jrplp\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564572 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fpzw7\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564576 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:20:21.565310 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.564661 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:20:21.574542 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.574504 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:20:21.574647 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.574560 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hxhjs\"" Oct 30 16:20:21.574715 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.574680 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lngbh\"" Oct 30 16:20:21.574773 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.574739 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:20:21.645994 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.645965 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-log-socket\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.646147 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646001 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-config\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.646147 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646028 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:21.646147 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-cnibin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.646147 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-multus\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.646147 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646079 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72976414-f225-493b-9c55-3ab89bb831a8-host\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.646147 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646097 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-log-socket\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.646147 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646100 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x66t\" (UniqueName: \"kubernetes.io/projected/569992a0-aa75-48cb-ad8d-fb61295a5093-kube-api-access-4x66t\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:21.646147 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-var-lib-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646157 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/be63e086-3851-459b-9123-262570be3b86-ovn-node-metrics-cert\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.646162 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646175 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn2zm\" (UniqueName: \"kubernetes.io/projected/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-kube-api-access-vn2zm\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646192 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-netns\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646206 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-multus-daemon-config\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.646233 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret podName:da3c94aa-c880-4844-8cc8-2182ff86e321 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:22.146211218 +0000 UTC m=+3.093593890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret") pod "global-pull-secret-syncer-26nhz" (UID: "da3c94aa-c880-4844-8cc8-2182ff86e321") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646265 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-systemd\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646294 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-host\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646321 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-etc-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646328 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-cnibin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646349 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646362 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-multus\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646376 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-env-overrides\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.646544 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646439 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72976414-f225-493b-9c55-3ab89bb831a8-host\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646633 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-config\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646675 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-multus-daemon-config\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646725 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-netns\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646728 2579 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646759 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-host\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646797 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-env-overrides\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646843 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646848 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-systemd\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-sys-fs\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646884 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646896 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646911 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-etc-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646918 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-run\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646943 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-tmp\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646946 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-sys-fs\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646963 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646975 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-var-lib-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.647277 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646941 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.646966 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647021 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb25da1d-2305-4f75-9920-e52fe596f5b7-hosts-file\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-run\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647051 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtgq\" (UniqueName: \"kubernetes.io/projected/cb25da1d-2305-4f75-9920-e52fe596f5b7-kube-api-access-cvtgq\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647060 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb25da1d-2305-4f75-9920-e52fe596f5b7-hosts-file\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647088 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647096 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647104 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12134b5c-de72-4972-a185-3d5450c7b5e7-agent-certs\") pod \"konnectivity-agent-glrfx\" (UID: \"12134b5c-de72-4972-a185-3d5450c7b5e7\") " pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647139 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72976414-f225-493b-9c55-3ab89bb831a8-serviceca\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647160 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-script-lib\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647212 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647220 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-system-cni-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647246 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647276 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-bin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.647285 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:21.648119 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647300 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-lib-modules\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.647331 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs podName:569992a0-aa75-48cb-ad8d-fb61295a5093 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:22.147315948 +0000 UTC m=+3.094698598 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs") pod "network-metrics-daemon-9ls8t" (UID: "569992a0-aa75-48cb-ad8d-fb61295a5093") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647346 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwsxf\" (UniqueName: \"kubernetes.io/projected/72976414-f225-493b-9c55-3ab89bb831a8-kube-api-access-bwsxf\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647369 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-netns\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647396 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-netd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647402 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-lib-modules\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647420 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647444 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12134b5c-de72-4972-a185-3d5450c7b5e7-konnectivity-ca\") pod \"konnectivity-agent-glrfx\" (UID: \"12134b5c-de72-4972-a185-3d5450c7b5e7\") " pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647462 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72976414-f225-493b-9c55-3ab89bb831a8-serviceca\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647471 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-system-cni-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647467 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-etc-tuned\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647537 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-system-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647567 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-var-lib-kubelet\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647592 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysconfig\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647637 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-sys\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e50ac2ed-9597-47ec-a4be-078d412f0b61-iptables-alerter-script\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.648992 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647671 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-bin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647689 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-dbus\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647721 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-netns\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647738 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-etc-kubernetes\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647764 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e50ac2ed-9597-47ec-a4be-078d412f0b61-host-slash\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647790 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-cni-binary-copy\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647812 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-conf-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647837 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-multus-certs\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647863 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25da1d-2305-4f75-9920-e52fe596f5b7-tmp-dir\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647887 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-k8s-cni-cncf-io\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647910 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-kubelet\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647934 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-registration-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647957 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-device-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647975 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-script-lib\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.647984 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-kubelet-config\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648009 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-socket-dir-parent\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.649805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648033 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-conf\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648040 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-netd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648060 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-slash\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648111 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-ovn\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648132 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-node-log\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648154 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds4gb\" (UniqueName: \"kubernetes.io/projected/be63e086-3851-459b-9123-262570be3b86-kube-api-access-ds4gb\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648181 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbpx\" (UniqueName: \"kubernetes.io/projected/cc9755b7-6248-49a7-93c2-06a74060d379-kube-api-access-qkbpx\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648227 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-kubelet\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648258 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-modprobe-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-kubernetes\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgv82\" (UniqueName: \"kubernetes.io/projected/e50ac2ed-9597-47ec-a4be-078d412f0b61-kube-api-access-sgv82\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648336 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648366 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648394 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw7vq\" (UniqueName: \"kubernetes.io/projected/5874f474-b442-4eec-a808-33997e342cb5-kube-api-access-lw7vq\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648422 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-systemd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648450 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-bin\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.653216 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648475 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cnibin\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648502 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqjj\" (UniqueName: \"kubernetes.io/projected/45b113ed-a156-4c53-bee5-26707e5717c3-kube-api-access-5hqjj\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648510 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-conf\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648549 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-socket-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648577 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-hostroot\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648582 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-conf-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-systemd-units\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648665 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-dbus\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648714 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648715 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-systemd-units\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648765 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-slash\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648814 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-ovn\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.648874 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-node-log\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649012 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-k8s-cni-cncf-io\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649072 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-kubelet\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649125 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-registration-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649174 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-device-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649220 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-kubelet-config\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:21.654124 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649262 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-socket-dir-parent\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649315 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12134b5c-de72-4972-a185-3d5450c7b5e7-konnectivity-ca\") pod \"konnectivity-agent-glrfx\" (UID: \"12134b5c-de72-4972-a185-3d5450c7b5e7\") " pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649360 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-kubelet\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649459 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-modprobe-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649515 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-kubernetes\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649594 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-multus-certs\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649681 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-bin\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649726 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.649795 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cnibin\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650154 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-systemd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650370 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25da1d-2305-4f75-9920-e52fe596f5b7-tmp-dir\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650491 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-hostroot\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650540 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-etc-kubernetes\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650581 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e50ac2ed-9597-47ec-a4be-078d412f0b61-host-slash\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650584 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e50ac2ed-9597-47ec-a4be-078d412f0b61-iptables-alerter-script\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650626 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-var-lib-kubelet\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650645 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-system-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650676 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysconfig\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.655026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650697 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-sys\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.655884 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650761 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-tmp\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.655884 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650771 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/be63e086-3851-459b-9123-262570be3b86-ovn-node-metrics-cert\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.655884 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650811 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-etc-tuned\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.655884 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650888 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-socket-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.655884 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.650900 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.655884 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.651053 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.655884 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.651090 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12134b5c-de72-4972-a185-3d5450c7b5e7-agent-certs\") pod \"konnectivity-agent-glrfx\" (UID: \"12134b5c-de72-4972-a185-3d5450c7b5e7\") " pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:21.655884 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.651111 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-cni-binary-copy\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.655884 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.651145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.670654 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.670528 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:21.670654 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.670548 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:21.670654 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.670561 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8kc9p for pod openshift-network-diagnostics/network-check-target-gccnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:21.670654 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:21.670638 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p podName:67d3a22c-930e-4113-ba61-2aecccabf9d2 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:22.170620545 +0000 UTC m=+3.118003213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-8kc9p" (UniqueName: "kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p") pod "network-check-target-gccnz" (UID: "67d3a22c-930e-4113-ba61-2aecccabf9d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:21.673251 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.672465 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn2zm\" (UniqueName: \"kubernetes.io/projected/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-kube-api-access-vn2zm\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.673251 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.672507 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x66t\" (UniqueName: \"kubernetes.io/projected/569992a0-aa75-48cb-ad8d-fb61295a5093-kube-api-access-4x66t\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:21.673839 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.673798 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtgq\" (UniqueName: \"kubernetes.io/projected/cb25da1d-2305-4f75-9920-e52fe596f5b7-kube-api-access-cvtgq\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.675053 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.675030 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwsxf\" (UniqueName: \"kubernetes.io/projected/72976414-f225-493b-9c55-3ab89bb831a8-kube-api-access-bwsxf\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.676007 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.675962 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqjj\" (UniqueName: \"kubernetes.io/projected/45b113ed-a156-4c53-bee5-26707e5717c3-kube-api-access-5hqjj\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.677991 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.677971 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw7vq\" (UniqueName: \"kubernetes.io/projected/5874f474-b442-4eec-a808-33997e342cb5-kube-api-access-lw7vq\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.678479 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.678443 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgv82\" (UniqueName: \"kubernetes.io/projected/e50ac2ed-9597-47ec-a4be-078d412f0b61-kube-api-access-sgv82\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.682233 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.682215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds4gb\" (UniqueName: \"kubernetes.io/projected/be63e086-3851-459b-9123-262570be3b86-kube-api-access-ds4gb\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.683916 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.683894 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbpx\" (UniqueName: \"kubernetes.io/projected/cc9755b7-6248-49a7-93c2-06a74060d379-kube-api-access-qkbpx\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.759569 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.759508 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:20:21.767929 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:21.767901 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5874f474_b442_4eec_a808_33997e342cb5.slice/crio-9f6c186de3fc25e0c4c4a9bf085c0360cb947cc3947870e55eda94c291a1f89b WatchSource:0}: Error finding container 9f6c186de3fc25e0c4c4a9bf085c0360cb947cc3947870e55eda94c291a1f89b: Status 404 returned error can't find the container with id 9f6c186de3fc25e0c4c4a9bf085c0360cb947cc3947870e55eda94c291a1f89b Oct 30 16:20:21.772507 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.772487 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:20:21.781844 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:21.781811 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9755b7_6248_49a7_93c2_06a74060d379.slice/crio-c480f118ec53bbba0e7858f55baeb2a3f1cf76b4ded3fd85b6d150ff40522aaa WatchSource:0}: Error finding container c480f118ec53bbba0e7858f55baeb2a3f1cf76b4ded3fd85b6d150ff40522aaa: Status 404 returned error can't find the container with id c480f118ec53bbba0e7858f55baeb2a3f1cf76b4ded3fd85b6d150ff40522aaa Oct 30 16:20:21.791500 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.791481 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:20:21.792573 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.792556 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-68psh" Oct 30 16:20:21.796336 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.796314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:20:21.803939 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.803920 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2bm4t" Oct 30 16:20:21.811694 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.811674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:20:21.812487 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:21.812464 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf4747b6_b5df_4e8e_a4f7_6de80acaad72.slice/crio-e068c96992dad3f8ef6035a37c77dd7ddf57157a437c388c4a5573b1bd53a9a9 WatchSource:0}: Error finding container e068c96992dad3f8ef6035a37c77dd7ddf57157a437c388c4a5573b1bd53a9a9: Status 404 returned error can't find the container with id e068c96992dad3f8ef6035a37c77dd7ddf57157a437c388c4a5573b1bd53a9a9 Oct 30 16:20:21.814934 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:21.814871 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72976414_f225_493b_9c55_3ab89bb831a8.slice/crio-a869f354c2abbc63e2aba5d9a42b0388132aa0f561bbfd6477624e707347ae4b WatchSource:0}: Error finding container a869f354c2abbc63e2aba5d9a42b0388132aa0f561bbfd6477624e707347ae4b: Status 404 returned error can't find the container with id a869f354c2abbc63e2aba5d9a42b0388132aa0f561bbfd6477624e707347ae4b Oct 30 16:20:21.819173 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:21.819148 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb25da1d_2305_4f75_9920_e52fe596f5b7.slice/crio-d0af2178deb53c5d43e8b86ff93686a2b479c30923bb0969a9ca457653639e35 WatchSource:0}: Error finding container d0af2178deb53c5d43e8b86ff93686a2b479c30923bb0969a9ca457653639e35: Status 404 returned error can't find the container with id d0af2178deb53c5d43e8b86ff93686a2b479c30923bb0969a9ca457653639e35 Oct 30 16:20:21.823088 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:21.822035 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45b113ed_a156_4c53_bee5_26707e5717c3.slice/crio-d8dd102e070f9a4d8557721bedc77be7ee8a47857446863ee6f9aea82af9ecdc WatchSource:0}: Error finding container d8dd102e070f9a4d8557721bedc77be7ee8a47857446863ee6f9aea82af9ecdc: Status 404 returned error can't find the container with id d8dd102e070f9a4d8557721bedc77be7ee8a47857446863ee6f9aea82af9ecdc Oct 30 16:20:21.826171 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:21.826108 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode50ac2ed_9597_47ec_a4be_078d412f0b61.slice/crio-09c1ac4d7705d3057cc77a8993d87c622e4d54efb2e60124b7819a6fa0f53fe3 WatchSource:0}: Error finding container 09c1ac4d7705d3057cc77a8993d87c622e4d54efb2e60124b7819a6fa0f53fe3: Status 404 returned error can't find the container with id 09c1ac4d7705d3057cc77a8993d87c622e4d54efb2e60124b7819a6fa0f53fe3 Oct 30 16:20:21.830361 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.830314 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:21.837742 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.837726 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:21.842593 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:21.842575 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12134b5c_de72_4972_a185_3d5450c7b5e7.slice/crio-8c8175c5ed0e3f758f1224bb024bd1c027e3efdc79d3a8f2b8d4516b86fcc475 WatchSource:0}: Error finding container 8c8175c5ed0e3f758f1224bb024bd1c027e3efdc79d3a8f2b8d4516b86fcc475: Status 404 returned error can't find the container with id 8c8175c5ed0e3f758f1224bb024bd1c027e3efdc79d3a8f2b8d4516b86fcc475 Oct 30 16:20:21.854760 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:20:21.854711 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe63e086_3851_459b_9123_262570be3b86.slice/crio-f696498181b3360832b434fed9f73d14e4a751195dcd8d9c3c64e7dd57ce6ffc WatchSource:0}: Error finding container f696498181b3360832b434fed9f73d14e4a751195dcd8d9c3c64e7dd57ce6ffc: Status 404 returned error can't find the container with id f696498181b3360832b434fed9f73d14e4a751195dcd8d9c3c64e7dd57ce6ffc Oct 30 16:20:21.952825 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:21.952793 2579 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:20:22.152388 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.152356 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:22.152537 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.152417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:22.152537 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:22.152532 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:22.152674 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:22.152532 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:22.152674 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:22.152588 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs podName:569992a0-aa75-48cb-ad8d-fb61295a5093 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:23.152569848 +0000 UTC m=+4.099952509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs") pod "network-metrics-daemon-9ls8t" (UID: "569992a0-aa75-48cb-ad8d-fb61295a5093") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:22.152780 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:22.152677 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret podName:da3c94aa-c880-4844-8cc8-2182ff86e321 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:23.152597049 +0000 UTC m=+4.099979695 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret") pod "global-pull-secret-syncer-26nhz" (UID: "da3c94aa-c880-4844-8cc8-2182ff86e321") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:22.253758 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.253721 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:22.254082 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:22.253949 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:22.254082 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:22.253981 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:22.254082 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:22.253997 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8kc9p for pod openshift-network-diagnostics/network-check-target-gccnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:22.254082 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:22.254063 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p podName:67d3a22c-930e-4113-ba61-2aecccabf9d2 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:23.254045288 +0000 UTC m=+4.201427935 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kc9p" (UniqueName: "kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p") pod "network-check-target-gccnz" (UID: "67d3a22c-930e-4113-ba61-2aecccabf9d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:22.495791 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.495575 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:15:21 +0000 UTC" deadline="2028-08-14 10:49:35.334886224 +0000 UTC" Oct 30 16:20:22.495791 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.495682 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15690h29m12.839209245s" Oct 30 16:20:22.605207 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.605076 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" event={"ID":"cf4747b6-b5df-4e8e-a4f7-6de80acaad72","Type":"ContainerStarted","Data":"e068c96992dad3f8ef6035a37c77dd7ddf57157a437c388c4a5573b1bd53a9a9"} Oct 30 16:20:22.615298 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.615224 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" event={"ID":"cc9755b7-6248-49a7-93c2-06a74060d379","Type":"ContainerStarted","Data":"c480f118ec53bbba0e7858f55baeb2a3f1cf76b4ded3fd85b6d150ff40522aaa"} Oct 30 16:20:22.633680 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.633631 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-772f2" event={"ID":"5874f474-b442-4eec-a808-33997e342cb5","Type":"ContainerStarted","Data":"9f6c186de3fc25e0c4c4a9bf085c0360cb947cc3947870e55eda94c291a1f89b"} Oct 30 16:20:22.637319 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.637234 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" event={"ID":"be63e086-3851-459b-9123-262570be3b86","Type":"ContainerStarted","Data":"f696498181b3360832b434fed9f73d14e4a751195dcd8d9c3c64e7dd57ce6ffc"} Oct 30 16:20:22.647477 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.647413 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-glrfx" event={"ID":"12134b5c-de72-4972-a185-3d5450c7b5e7","Type":"ContainerStarted","Data":"8c8175c5ed0e3f758f1224bb024bd1c027e3efdc79d3a8f2b8d4516b86fcc475"} Oct 30 16:20:22.659627 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.659584 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p528z" event={"ID":"e50ac2ed-9597-47ec-a4be-078d412f0b61","Type":"ContainerStarted","Data":"09c1ac4d7705d3057cc77a8993d87c622e4d54efb2e60124b7819a6fa0f53fe3"} Oct 30 16:20:22.672702 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.672675 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2bm4t" event={"ID":"45b113ed-a156-4c53-bee5-26707e5717c3","Type":"ContainerStarted","Data":"d8dd102e070f9a4d8557721bedc77be7ee8a47857446863ee6f9aea82af9ecdc"} Oct 30 16:20:22.679829 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.679153 2579 generic.go:356] "Generic (PLEG): container finished" podID="c459e3b80b3dfd1e7e8df0c149b8ceb5" containerID="39adf7c56cacde2be481386b6435f0530ef499db9267370a85ddfad6c380ec3f" exitCode=0 Oct 30 16:20:22.679829 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.679240 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" event={"ID":"c459e3b80b3dfd1e7e8df0c149b8ceb5","Type":"ContainerDied","Data":"39adf7c56cacde2be481386b6435f0530ef499db9267370a85ddfad6c380ec3f"} Oct 30 16:20:22.692352 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.691920 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" event={"ID":"3c2020069c5384399725929d6ee8319e","Type":"ContainerStarted","Data":"f0cbbf21b46508a43f980d555347c6ea4ee62a01dd645df91ff681c6de1dc28c"} Oct 30 16:20:22.694534 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.694479 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-68psh" event={"ID":"cb25da1d-2305-4f75-9920-e52fe596f5b7","Type":"ContainerStarted","Data":"d0af2178deb53c5d43e8b86ff93686a2b479c30923bb0969a9ca457653639e35"} Oct 30 16:20:22.701331 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.701274 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rns4g" event={"ID":"72976414-f225-493b-9c55-3ab89bb831a8","Type":"ContainerStarted","Data":"a869f354c2abbc63e2aba5d9a42b0388132aa0f561bbfd6477624e707347ae4b"} Oct 30 16:20:22.763634 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:22.763526 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" podStartSLOduration=2.763508678 podStartE2EDuration="2.763508678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:22.762864043 +0000 UTC m=+3.710246712" watchObservedRunningTime="2026-10-30 16:20:22.763508678 +0000 UTC m=+3.710891346" Oct 30 16:20:23.160886 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:23.160830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:23.161066 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:23.160909 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:23.161066 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.161054 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:23.161179 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.161111 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs podName:569992a0-aa75-48cb-ad8d-fb61295a5093 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:25.161093509 +0000 UTC m=+6.108476157 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs") pod "network-metrics-daemon-9ls8t" (UID: "569992a0-aa75-48cb-ad8d-fb61295a5093") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:23.161540 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.161524 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:23.161627 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.161577 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret podName:da3c94aa-c880-4844-8cc8-2182ff86e321 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:25.161562852 +0000 UTC m=+6.108945499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret") pod "global-pull-secret-syncer-26nhz" (UID: "da3c94aa-c880-4844-8cc8-2182ff86e321") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:23.261928 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:23.261891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:23.262085 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.262058 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:23.262085 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.262077 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:23.262197 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.262089 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8kc9p for pod openshift-network-diagnostics/network-check-target-gccnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:23.262197 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.262144 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p podName:67d3a22c-930e-4113-ba61-2aecccabf9d2 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:25.262127732 +0000 UTC m=+6.209510391 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kc9p" (UniqueName: "kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p") pod "network-check-target-gccnz" (UID: "67d3a22c-930e-4113-ba61-2aecccabf9d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:23.496895 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:23.496839 2579 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:15:21 +0000 UTC" deadline="2028-04-15 04:12:36.480113257 +0000 UTC" Oct 30 16:20:23.496895 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:23.496895 2579 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12779h52m12.983224904s" Oct 30 16:20:23.597880 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:23.597363 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:23.597880 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.597464 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:23.597880 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:23.597828 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:23.598139 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.597920 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:23.598139 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:23.597974 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:23.598139 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:23.598039 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:23.732044 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:23.731856 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" event={"ID":"c459e3b80b3dfd1e7e8df0c149b8ceb5","Type":"ContainerStarted","Data":"2a721d5ebbeab1cd2c948704c44167af366ad2c5654b9fffe18419a866a8c583"} Oct 30 16:20:24.743016 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:24.741769 2579 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="ee5acd6570feb185531357147ad8c37ea41f1d7043f5e9693048033544c6f9fd" exitCode=0 Oct 30 16:20:24.743016 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:24.742763 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" event={"ID":"cf4747b6-b5df-4e8e-a4f7-6de80acaad72","Type":"ContainerDied","Data":"ee5acd6570feb185531357147ad8c37ea41f1d7043f5e9693048033544c6f9fd"} Oct 30 16:20:24.813176 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:24.813121 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" podStartSLOduration=4.813102248 podStartE2EDuration="4.813102248s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:23.772431614 +0000 UTC m=+4.719814284" watchObservedRunningTime="2026-10-30 16:20:24.813102248 +0000 UTC m=+5.760484917" Oct 30 16:20:25.177159 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:25.177092 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:25.177302 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:25.177182 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:25.177302 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.177289 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:25.177406 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.177344 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret podName:da3c94aa-c880-4844-8cc8-2182ff86e321 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:29.17732782 +0000 UTC m=+10.124710479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret") pod "global-pull-secret-syncer-26nhz" (UID: "da3c94aa-c880-4844-8cc8-2182ff86e321") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:25.177805 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.177725 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:25.177805 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.177777 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs podName:569992a0-aa75-48cb-ad8d-fb61295a5093 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:29.177762592 +0000 UTC m=+10.125145240 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs") pod "network-metrics-daemon-9ls8t" (UID: "569992a0-aa75-48cb-ad8d-fb61295a5093") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:25.278265 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:25.277726 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:25.278265 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.277869 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:25.278265 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.277885 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:25.278265 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.277895 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8kc9p for pod openshift-network-diagnostics/network-check-target-gccnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:25.278265 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.277942 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p podName:67d3a22c-930e-4113-ba61-2aecccabf9d2 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:29.277928591 +0000 UTC m=+10.225311253 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kc9p" (UniqueName: "kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p") pod "network-check-target-gccnz" (UID: "67d3a22c-930e-4113-ba61-2aecccabf9d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:25.594071 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:25.594006 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:25.594233 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.594144 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:25.594529 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:25.594513 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:25.594757 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:25.594714 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:25.594841 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.594816 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:25.594947 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:25.594829 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:26.752501 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:26.751976 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-glrfx" event={"ID":"12134b5c-de72-4972-a185-3d5450c7b5e7","Type":"ContainerStarted","Data":"db19cf1192b460369fc9186699f6bbc061768d3e8eed579a458b30f8355737fd"} Oct 30 16:20:26.794853 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:26.794802 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-glrfx" podStartSLOduration=3.997607642 podStartE2EDuration="7.794788853s" totalImagesPullingTime="3.797181211s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:26.793412517 +0000 UTC m=+7.740795184" watchObservedRunningTime="2026-10-30 16:20:26.794788853 +0000 UTC m=+7.742171521" Oct 30 16:20:26.976629 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:26.976169 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:26.977396 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:26.977203 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:27.591856 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:27.591827 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:27.592012 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:27.591930 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:27.592178 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:27.592163 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:27.592256 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:27.592239 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:27.592312 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:27.592288 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:27.592373 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:27.592328 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:27.763292 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:27.763259 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-rns4g" event={"ID":"72976414-f225-493b-9c55-3ab89bb831a8","Type":"ContainerStarted","Data":"743b5a975d5bef10398b1c1b939541dab8ab2713dc7d0a85eebf7dce734df2e1"} Oct 30 16:20:28.771805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:28.771774 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" event={"ID":"cc9755b7-6248-49a7-93c2-06a74060d379","Type":"ContainerStarted","Data":"347c49bff56aa28690815af25183d63431e97494c3806a08c788911126e89051"} Oct 30 16:20:28.772254 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:28.771870 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:29.214583 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.214550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:29.214757 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.214620 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:29.214832 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.214760 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:29.214832 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.214816 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs podName:569992a0-aa75-48cb-ad8d-fb61295a5093 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.214797814 +0000 UTC m=+18.162180460 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs") pod "network-metrics-daemon-9ls8t" (UID: "569992a0-aa75-48cb-ad8d-fb61295a5093") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:29.215228 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.215207 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:29.215346 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.215261 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret podName:da3c94aa-c880-4844-8cc8-2182ff86e321 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.215247373 +0000 UTC m=+18.162630031 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret") pod "global-pull-secret-syncer-26nhz" (UID: "da3c94aa-c880-4844-8cc8-2182ff86e321") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:29.306867 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.306805 2579 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:20:29.315214 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.315186 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:29.315361 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.315299 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:29.315361 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.315313 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:29.315361 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.315323 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8kc9p for pod openshift-network-diagnostics/network-check-target-gccnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:29.315503 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.315369 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p podName:67d3a22c-930e-4113-ba61-2aecccabf9d2 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:37.31535613 +0000 UTC m=+18.262738775 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kc9p" (UniqueName: "kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p") pod "network-check-target-gccnz" (UID: "67d3a22c-930e-4113-ba61-2aecccabf9d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:29.509682 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.508998 2579 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:20:29.306822115Z","UUID":"5bb7f7db-30ff-4778-831d-f4b65cb6f22a","Handler":null,"Name":"","Endpoint":""} Oct 30 16:20:29.510900 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.510876 2579 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:20:29.511026 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.510909 2579 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:20:29.592518 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.592483 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:29.592666 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.592572 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:29.592741 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.592683 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:29.592844 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.592784 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:29.592844 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.592790 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:29.592964 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:29.592877 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:29.775312 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.775277 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2bm4t" event={"ID":"45b113ed-a156-4c53-bee5-26707e5717c3","Type":"ContainerStarted","Data":"2c395f515455b4b36ee7979cf12ea820bbbd60ee8d73ff919c6aba6680de7882"} Oct 30 16:20:29.779719 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.779692 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-68psh" event={"ID":"cb25da1d-2305-4f75-9920-e52fe596f5b7","Type":"ContainerStarted","Data":"79b5d8feb06b58065306d2043144671852a12058ff4a67710ec0b8e3864f4378"} Oct 30 16:20:29.781986 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.781960 2579 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="bb1e64256263c66b4f519130128df8744a7dd9fe28181a078906be19129a7a51" exitCode=0 Oct 30 16:20:29.782074 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.782040 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" event={"ID":"cf4747b6-b5df-4e8e-a4f7-6de80acaad72","Type":"ContainerDied","Data":"bb1e64256263c66b4f519130128df8744a7dd9fe28181a078906be19129a7a51"} Oct 30 16:20:29.784364 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.784342 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" event={"ID":"cc9755b7-6248-49a7-93c2-06a74060d379","Type":"ContainerStarted","Data":"71d063bec7f098bcaeece5c2f25d022f703a3b4e7f72f506193bf888c8ac616b"} Oct 30 16:20:29.820977 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.820931 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-rns4g" podStartSLOduration=5.872830662 podStartE2EDuration="10.820917471s" totalImagesPullingTime="4.948086809s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:27.805593947 +0000 UTC m=+8.752976613" watchObservedRunningTime="2026-10-30 16:20:29.820917471 +0000 UTC m=+10.768300138" Oct 30 16:20:29.821108 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.821030 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-2bm4t" podStartSLOduration=3.59647383 podStartE2EDuration="10.821025483s" totalImagesPullingTime="7.224551653s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:29.82050219 +0000 UTC m=+10.767884857" watchObservedRunningTime="2026-10-30 16:20:29.821025483 +0000 UTC m=+10.768408236" Oct 30 16:20:29.863081 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:29.862981 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-68psh" podStartSLOduration=3.050243971 podStartE2EDuration="10.862969149s" totalImagesPullingTime="7.812725178s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:29.862220564 +0000 UTC m=+10.809603231" watchObservedRunningTime="2026-10-30 16:20:29.862969149 +0000 UTC m=+10.810351819" Oct 30 16:20:30.790167 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:30.789949 2579 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="ba415ee5db4580a713708763a9fb6ca6432a02a653a9b75f3baa81f8d5cb347f" exitCode=0 Oct 30 16:20:30.790577 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:30.790052 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" event={"ID":"cf4747b6-b5df-4e8e-a4f7-6de80acaad72","Type":"ContainerDied","Data":"ba415ee5db4580a713708763a9fb6ca6432a02a653a9b75f3baa81f8d5cb347f"} Oct 30 16:20:30.794155 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:30.794131 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" event={"ID":"cc9755b7-6248-49a7-93c2-06a74060d379","Type":"ContainerStarted","Data":"199bd27e1aeaec9b645f1e54b5225130ad8ba8dee4c39118c4690ea213acd096"} Oct 30 16:20:30.795958 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:30.795898 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-p528z" event={"ID":"e50ac2ed-9597-47ec-a4be-078d412f0b61","Type":"ContainerStarted","Data":"29e60fce23094e6bdd48112730d0ca0587e84a2ebb0575fa55439429b959adfa"} Oct 30 16:20:30.931811 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:30.931740 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-p528z" podStartSLOduration=4.415040568 podStartE2EDuration="11.931720058s" totalImagesPullingTime="7.51667949s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:30.883628619 +0000 UTC m=+11.831011290" watchObservedRunningTime="2026-10-30 16:20:30.931720058 +0000 UTC m=+11.879102721" Oct 30 16:20:30.932127 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:30.932100 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" podStartSLOduration=3.704575706 podStartE2EDuration="11.932090826s" totalImagesPullingTime="8.22751512s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:30.931613793 +0000 UTC m=+11.878996455" watchObservedRunningTime="2026-10-30 16:20:30.932090826 +0000 UTC m=+11.879473497" Oct 30 16:20:31.592347 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:31.592325 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:31.592472 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:31.592447 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:31.592796 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:31.592775 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:31.592897 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:31.592882 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:31.593648 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:31.592276 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:31.593648 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:31.593539 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:31.800317 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:31.800285 2579 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="0b75f64d87246ad1b8a5452226c9838caf0f782eed4d45808ee4b96747624e89" exitCode=0 Oct 30 16:20:31.801009 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:31.800375 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" event={"ID":"cf4747b6-b5df-4e8e-a4f7-6de80acaad72","Type":"ContainerDied","Data":"0b75f64d87246ad1b8a5452226c9838caf0f782eed4d45808ee4b96747624e89"} Oct 30 16:20:31.802365 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:31.802340 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-772f2" event={"ID":"5874f474-b442-4eec-a808-33997e342cb5","Type":"ContainerStarted","Data":"97ade5e0cf955bce5f79b27d55ef6f3a56975e0f98853017b8bcfdda6561024d"} Oct 30 16:20:31.897488 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:31.897398 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-772f2" podStartSLOduration=3.107310581 podStartE2EDuration="12.897382698s" totalImagesPullingTime="9.790072117s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:31.897142242 +0000 UTC m=+12.844524908" watchObservedRunningTime="2026-10-30 16:20:31.897382698 +0000 UTC m=+12.844765364" Oct 30 16:20:33.591671 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:33.591634 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:33.592349 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:33.591639 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:33.592349 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:33.591770 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:33.592349 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:33.591774 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:33.592349 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:33.591838 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:33.592349 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:33.591987 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:33.809829 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:33.809794 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" event={"ID":"cf4747b6-b5df-4e8e-a4f7-6de80acaad72","Type":"ContainerStarted","Data":"b3c19736369e5e211535094ac95ba764667fe5e2372c442dccb19a8e93167f64"} Oct 30 16:20:34.723057 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:34.722816 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:34.723509 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:34.723168 2579 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 16:20:34.723570 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:34.723512 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:20:34.815244 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:34.815211 2579 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="b3c19736369e5e211535094ac95ba764667fe5e2372c442dccb19a8e93167f64" exitCode=0 Oct 30 16:20:34.815395 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:34.815267 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" event={"ID":"cf4747b6-b5df-4e8e-a4f7-6de80acaad72","Type":"ContainerDied","Data":"b3c19736369e5e211535094ac95ba764667fe5e2372c442dccb19a8e93167f64"} Oct 30 16:20:35.591897 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:35.591866 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:35.591897 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:35.591887 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:35.592091 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:35.591867 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:35.592091 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:35.591967 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:35.592091 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:35.592048 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:35.592226 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:35.592129 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:35.819487 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:35.819456 2579 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="c9d20295d1fbe7cf1d0bfbd1a8cfe0cc183043297b8690d6afbf8623e252cc0b" exitCode=0 Oct 30 16:20:35.819978 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:35.819516 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" event={"ID":"cf4747b6-b5df-4e8e-a4f7-6de80acaad72","Type":"ContainerDied","Data":"c9d20295d1fbe7cf1d0bfbd1a8cfe0cc183043297b8690d6afbf8623e252cc0b"} Oct 30 16:20:36.825466 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:36.825210 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" event={"ID":"cf4747b6-b5df-4e8e-a4f7-6de80acaad72","Type":"ContainerStarted","Data":"274b599fff3bff6429ae4566c242495a4db9cdfad97bb9139b7eefa86d3bade2"} Oct 30 16:20:36.828071 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:36.828044 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" event={"ID":"be63e086-3851-459b-9123-262570be3b86","Type":"ContainerStarted","Data":"a2131782f7844bb0609be3347a72f58035f97ca282d4775a3d46ce9a01a739be"} Oct 30 16:20:36.828165 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:36.828105 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" event={"ID":"be63e086-3851-459b-9123-262570be3b86","Type":"ContainerStarted","Data":"0d53bcc8e5b6162421e48b4b5344c9ee797cf2968bc17a6d9ef2cebafae1de5a"} Oct 30 16:20:36.828165 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:36.828143 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" event={"ID":"be63e086-3851-459b-9123-262570be3b86","Type":"ContainerStarted","Data":"ae31a400bf3c102d8b277e05933655cc3a1a1f930bebf34bd4b1b260900e9de3"} Oct 30 16:20:36.828222 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:36.828170 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" event={"ID":"be63e086-3851-459b-9123-262570be3b86","Type":"ContainerStarted","Data":"a60a82094082440996fbac3247538dd0e1078e9c5dd555c4a4be2c501b93dc80"} Oct 30 16:20:36.828222 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:36.828200 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" event={"ID":"be63e086-3851-459b-9123-262570be3b86","Type":"ContainerStarted","Data":"4e35fc09aa3eec41d05699b08a1a75397676046f6ecdfc4e89e52a333e702894"} Oct 30 16:20:36.828293 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:36.828229 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" event={"ID":"be63e086-3851-459b-9123-262570be3b86","Type":"ContainerStarted","Data":"f9fcc01122749b3901e39ce69c480dc8e789b7ec8a8ccfb4270e7d9d407f71a5"} Oct 30 16:20:36.875156 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:36.875116 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n8mr5" podStartSLOduration=7.931652744 podStartE2EDuration="17.87510337s" totalImagesPullingTime="9.943450626s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:36.874532885 +0000 UTC m=+17.821915552" watchObservedRunningTime="2026-10-30 16:20:36.87510337 +0000 UTC m=+17.822486037" Oct 30 16:20:37.284787 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:37.284713 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:37.284787 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:37.284753 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:37.284958 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.284842 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:37.284958 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.284843 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:37.284958 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.284898 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret podName:da3c94aa-c880-4844-8cc8-2182ff86e321 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:53.284884516 +0000 UTC m=+34.232267162 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret") pod "global-pull-secret-syncer-26nhz" (UID: "da3c94aa-c880-4844-8cc8-2182ff86e321") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:37.284958 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.284911 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs podName:569992a0-aa75-48cb-ad8d-fb61295a5093 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:53.284905472 +0000 UTC m=+34.232288117 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs") pod "network-metrics-daemon-9ls8t" (UID: "569992a0-aa75-48cb-ad8d-fb61295a5093") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:37.385984 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:37.385955 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:37.386111 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.386097 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:37.386151 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.386113 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:37.386151 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.386122 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8kc9p for pod openshift-network-diagnostics/network-check-target-gccnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:37.386205 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.386167 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p podName:67d3a22c-930e-4113-ba61-2aecccabf9d2 nodeName:}" failed. No retries permitted until 2026-10-30 16:20:53.386152036 +0000 UTC m=+34.333534683 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kc9p" (UniqueName: "kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p") pod "network-check-target-gccnz" (UID: "67d3a22c-930e-4113-ba61-2aecccabf9d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:37.591724 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:37.591655 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:37.591724 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:37.591670 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:37.591724 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:37.591676 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:37.591913 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.591755 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:37.591913 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.591829 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:37.591913 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:37.591902 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:38.834546 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:38.834464 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" event={"ID":"be63e086-3851-459b-9123-262570be3b86","Type":"ContainerStarted","Data":"74cdd94456c3238b5d57670a6a0c9db957b6b7db8c4cc64099f6fdbf4c0ef79a"} Oct 30 16:20:39.591781 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:39.591751 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:39.591937 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:39.591844 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:39.591937 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:39.591908 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:39.592052 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:39.591979 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:39.592052 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:39.591985 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:39.592052 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:39.592041 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:40.841373 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:40.841293 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" event={"ID":"be63e086-3851-459b-9123-262570be3b86","Type":"ContainerStarted","Data":"161c94a6c5212ed624fc301d8f809d0a225dfb8e14feccae0d44312c139a8f74"} Oct 30 16:20:40.842193 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:40.841636 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:40.842193 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:40.841708 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:40.842193 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:40.841754 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:40.857349 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:40.857324 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:40.857469 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:40.857422 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:20:40.947446 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:40.947405 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" podStartSLOduration=7.767622334 podStartE2EDuration="21.947394718s" totalImagesPullingTime="14.179772384s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:20:40.933621902 +0000 UTC m=+21.881004569" watchObservedRunningTime="2026-10-30 16:20:40.947394718 +0000 UTC m=+21.894777384" Oct 30 16:20:41.591973 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:41.591938 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:41.591973 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:41.591970 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:41.592189 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:41.592049 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:41.592189 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:41.592097 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:41.592189 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:41.592170 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:41.592332 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:41.592220 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:42.041317 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:42.041254 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-68psh_cb25da1d-2305-4f75-9920-e52fe596f5b7/dns-node-resolver/0.log" Oct 30 16:20:43.293989 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:43.293958 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rns4g_72976414-f225-493b-9c55-3ab89bb831a8/node-ca/0.log" Oct 30 16:20:43.591793 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:43.591721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:43.591793 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:43.591755 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:43.591954 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:43.591823 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:43.591954 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:43.591856 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:43.591954 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:43.591934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:43.592061 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:43.591987 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:45.591934 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:45.591904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:45.591934 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:45.591927 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:45.592427 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:45.591904 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:45.592427 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:45.592015 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:45.592427 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:45.592059 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:45.592427 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:45.592117 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:47.591887 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:47.591859 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:47.591887 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:47.591876 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:47.592391 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:47.591860 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:47.592391 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:47.591956 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:47.592391 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:47.592027 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:47.592391 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:47.592128 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:49.592438 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:49.592254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:49.592913 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:49.592345 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:49.592913 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:49.592515 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:49.592913 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:49.592377 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:49.592913 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:49.592635 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:49.592913 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:49.592666 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:51.591815 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:51.591779 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:51.592188 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:51.591788 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:51.592188 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:51.591917 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:51.592188 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:51.591895 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:51.592188 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:51.591993 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:51.592188 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:51.592079 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:53.302419 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:53.302388 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:53.302792 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:53.302438 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:53.302792 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.302517 2579 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:53.302792 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.302526 2579 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:53.302792 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.302571 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs podName:569992a0-aa75-48cb-ad8d-fb61295a5093 nodeName:}" failed. No retries permitted until 2026-10-30 16:21:25.302557549 +0000 UTC m=+66.249940195 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs") pod "network-metrics-daemon-9ls8t" (UID: "569992a0-aa75-48cb-ad8d-fb61295a5093") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 16:20:53.302792 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.302584 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret podName:da3c94aa-c880-4844-8cc8-2182ff86e321 nodeName:}" failed. No retries permitted until 2026-10-30 16:21:25.302578231 +0000 UTC m=+66.249960875 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret") pod "global-pull-secret-syncer-26nhz" (UID: "da3c94aa-c880-4844-8cc8-2182ff86e321") : object "kube-system"/"original-pull-secret" not registered Oct 30 16:20:53.403225 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:53.403193 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:53.403380 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.403306 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 16:20:53.403380 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.403319 2579 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 16:20:53.403380 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.403328 2579 projected.go:196] Error preparing data for projected volume kube-api-access-8kc9p for pod openshift-network-diagnostics/network-check-target-gccnz: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:53.403380 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.403369 2579 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p podName:67d3a22c-930e-4113-ba61-2aecccabf9d2 nodeName:}" failed. No retries permitted until 2026-10-30 16:21:25.403358027 +0000 UTC m=+66.350740685 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-8kc9p" (UniqueName: "kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p") pod "network-check-target-gccnz" (UID: "67d3a22c-930e-4113-ba61-2aecccabf9d2") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 16:20:53.591545 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:53.591477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:53.591545 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:53.591490 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:53.591746 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:53.591477 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:53.591746 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.591582 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:53.591746 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.591691 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:53.591859 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:53.591783 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:55.591565 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:55.591539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:55.591565 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:55.591554 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:55.591972 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:55.591539 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:55.591972 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:55.591656 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:55.591972 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:55.591729 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:55.591972 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:55.591800 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:57.591750 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:57.591715 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:57.591750 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:57.591729 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:57.592190 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:57.591863 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:57.592190 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:57.591917 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:57.592190 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:57.592029 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:57.592190 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:57.592082 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:20:59.592957 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:59.592924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:20:59.593392 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:59.593085 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:20:59.593392 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:59.593147 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:20:59.593392 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:59.593243 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:20:59.593644 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:20:59.593624 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:20:59.593772 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:20:59.593753 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:21:01.591874 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:01.591843 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:01.591874 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:01.591855 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:01.591874 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:01.591858 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:01.592397 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:01.591934 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:21:01.592397 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:01.592036 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:21:01.592397 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:01.592080 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:21:03.593657 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:03.593626 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:03.594129 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:03.593761 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:21:03.594129 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:03.593901 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:03.594129 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:03.594002 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:21:03.594290 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:03.594193 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:03.594341 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:03.594284 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:21:03.595944 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:03.595925 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gccnz"] Oct 30 16:21:03.598346 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:03.598323 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-26nhz"] Oct 30 16:21:03.598524 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:03.598510 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9ls8t"] Oct 30 16:21:03.886315 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:03.886247 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:03.886460 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:03.886254 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:03.886460 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:03.886344 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:21:03.886460 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:03.886252 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:03.886460 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:03.886418 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:21:03.886708 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:03.886520 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:21:03.928703 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:03.928685 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-68psh_cb25da1d-2305-4f75-9920-e52fe596f5b7/dns-node-resolver/0.log" Oct 30 16:21:05.529320 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:05.529295 2579 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rns4g_72976414-f225-493b-9c55-3ab89bb831a8/node-ca/0.log" Oct 30 16:21:05.591872 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:05.591840 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:05.592019 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:05.591890 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:05.592019 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:05.591988 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:05.592019 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:05.591993 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:21:05.592190 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:05.592123 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:21:05.592190 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:05.592166 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:21:07.592049 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.592019 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:07.592452 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.592018 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:07.592452 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:07.592116 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-gccnz" podUID="67d3a22c-930e-4113-ba61-2aecccabf9d2" Oct 30 16:21:07.592452 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:07.592182 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-26nhz" podUID="da3c94aa-c880-4844-8cc8-2182ff86e321" Oct 30 16:21:07.592452 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.592026 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:07.592452 ip-10-0-128-181 kubenswrapper[2579]: E1030 16:21:07.592251 2579 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-9ls8t" podUID="569992a0-aa75-48cb-ad8d-fb61295a5093" Oct 30 16:21:07.776476 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.776445 2579 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeReady" Oct 30 16:21:07.776671 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.776594 2579 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 16:21:07.839700 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.839666 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6qzkb"] Oct 30 16:21:07.841929 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.841906 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:07.845741 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.843387 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qhm42"] Oct 30 16:21:07.846353 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.846332 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:07.858665 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.858646 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:21:07.858755 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.858737 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:21:07.858755 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.858737 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6q4zp\"" Oct 30 16:21:07.858836 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.858771 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:21:07.858836 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.858777 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:21:07.864487 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.864469 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6qzkb"] Oct 30 16:21:07.872733 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.872714 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qhm42"] Oct 30 16:21:07.945645 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.945591 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-v998h"] Oct 30 16:21:07.947466 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.947452 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:07.958825 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.958805 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s8kck\"" Oct 30 16:21:07.958937 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.958864 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:21:07.958937 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.958878 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:21:07.963733 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.963712 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:21:07.963839 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.963826 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:21:07.963885 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.963832 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:21:07.963923 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.963910 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m9dmv\"" Oct 30 16:21:07.969294 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:07.969270 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v998h"] Oct 30 16:21:08.008491 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.008458 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.008491 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.008487 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8e15255-91bb-411a-9c47-3c714735e70d-metrics-tls\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.008712 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.008508 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gnnr\" (UniqueName: \"kubernetes.io/projected/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-api-access-2gnnr\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.008712 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.008526 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-crio-socket\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.008712 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.008633 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v87r5\" (UniqueName: \"kubernetes.io/projected/c8e15255-91bb-411a-9c47-3c714735e70d-kube-api-access-v87r5\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.008712 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.008660 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-data-volume\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.008712 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.008678 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e15255-91bb-411a-9c47-3c714735e70d-tmp-dir\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.008867 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.008734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.008867 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.008824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e15255-91bb-411a-9c47-3c714735e70d-config-volume\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.109908 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.109824 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-cert\") pod \"ingress-canary-v998h\" (UID: \"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a\") " pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:08.109908 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.109862 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwrf\" (UniqueName: \"kubernetes.io/projected/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-kube-api-access-ldwrf\") pod \"ingress-canary-v998h\" (UID: \"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a\") " pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:08.109908 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.109891 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.110176 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.109970 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e15255-91bb-411a-9c47-3c714735e70d-config-volume\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.110176 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110025 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.110176 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110042 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8e15255-91bb-411a-9c47-3c714735e70d-metrics-tls\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.110176 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110061 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gnnr\" (UniqueName: \"kubernetes.io/projected/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-api-access-2gnnr\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.110176 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110077 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-crio-socket\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.110176 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110097 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v87r5\" (UniqueName: \"kubernetes.io/projected/c8e15255-91bb-411a-9c47-3c714735e70d-kube-api-access-v87r5\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.110426 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110256 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-data-volume\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.110426 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110287 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e15255-91bb-411a-9c47-3c714735e70d-tmp-dir\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.110426 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110398 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-crio-socket\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.110738 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110716 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-data-volume\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.110813 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110719 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e15255-91bb-411a-9c47-3c714735e70d-tmp-dir\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.110872 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110820 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.110872 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.110832 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e15255-91bb-411a-9c47-3c714735e70d-config-volume\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.113983 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.113965 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8e15255-91bb-411a-9c47-3c714735e70d-metrics-tls\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.113983 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.113972 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.135345 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.135313 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gnnr\" (UniqueName: \"kubernetes.io/projected/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-api-access-2gnnr\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.135437 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.135416 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v87r5\" (UniqueName: \"kubernetes.io/projected/c8e15255-91bb-411a-9c47-3c714735e70d-kube-api-access-v87r5\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.157265 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.157241 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:08.161144 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.161129 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:08.211077 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.211046 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-cert\") pod \"ingress-canary-v998h\" (UID: \"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a\") " pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:08.211254 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.211084 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwrf\" (UniqueName: \"kubernetes.io/projected/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-kube-api-access-ldwrf\") pod \"ingress-canary-v998h\" (UID: \"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a\") " pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:08.214121 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.214090 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-cert\") pod \"ingress-canary-v998h\" (UID: \"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a\") " pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:08.237365 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.237340 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwrf\" (UniqueName: \"kubernetes.io/projected/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-kube-api-access-ldwrf\") pod \"ingress-canary-v998h\" (UID: \"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a\") " pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:08.257083 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.257059 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:08.375813 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.375744 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-c966567fd-rv9sr"] Oct 30 16:21:08.377722 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.377708 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.394503 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.394472 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 16:21:08.394503 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.394491 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 16:21:08.394713 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.394483 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 16:21:08.394713 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.394634 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 16:21:08.394713 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.394703 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 16:21:08.394846 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.394718 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 16:21:08.394846 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.394754 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 16:21:08.394846 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.394779 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lbtjk\"" Oct 30 16:21:08.402038 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.402018 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c966567fd-rv9sr"] Oct 30 16:21:08.465136 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.465111 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qhm42"] Oct 30 16:21:08.466977 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:08.466953 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8e15255_91bb_411a_9c47_3c714735e70d.slice/crio-55e377cd2543dde7f785cffa2dbd8e0a59f376979a1b06d430400698f1e51f5b WatchSource:0}: Error finding container 55e377cd2543dde7f785cffa2dbd8e0a59f376979a1b06d430400698f1e51f5b: Status 404 returned error can't find the container with id 55e377cd2543dde7f785cffa2dbd8e0a59f376979a1b06d430400698f1e51f5b Oct 30 16:21:08.476559 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.476532 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6qzkb"] Oct 30 16:21:08.477592 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:08.477569 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1dc93b8_6bd0_484e_a06c_d5a659e44eba.slice/crio-07d57a9af5deb83e3e36e68a621dec198a98234087076fcea3f17542366a4eb5 WatchSource:0}: Error finding container 07d57a9af5deb83e3e36e68a621dec198a98234087076fcea3f17542366a4eb5: Status 404 returned error can't find the container with id 07d57a9af5deb83e3e36e68a621dec198a98234087076fcea3f17542366a4eb5 Oct 30 16:21:08.513934 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.513910 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2dv7\" (UniqueName: \"kubernetes.io/projected/e22030b7-c861-47ee-b7d7-75f6d659e9c2-kube-api-access-j2dv7\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.514031 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.514016 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-serving-cert\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.514068 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.514048 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-service-ca\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.514111 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.514097 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22030b7-c861-47ee-b7d7-75f6d659e9c2-tmp\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.514149 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.514123 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-oauth-config\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.514214 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.514190 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-config\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.514246 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.514233 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-oauth-serving-cert\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.546745 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.546725 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-v998h"] Oct 30 16:21:08.548814 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:08.548784 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd1218c_9c3d_4d82_81f2_3cf601ebeb9a.slice/crio-5bc7b0b7b31bf53d7846f8be2ee9ffd5caa807beed7aa0c61800119b74253beb WatchSource:0}: Error finding container 5bc7b0b7b31bf53d7846f8be2ee9ffd5caa807beed7aa0c61800119b74253beb: Status 404 returned error can't find the container with id 5bc7b0b7b31bf53d7846f8be2ee9ffd5caa807beed7aa0c61800119b74253beb Oct 30 16:21:08.614826 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.614799 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-service-ca\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.614826 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.614829 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22030b7-c861-47ee-b7d7-75f6d659e9c2-tmp\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.615331 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.614852 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-oauth-config\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.615331 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.614871 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-config\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.615331 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.614893 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-oauth-serving-cert\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.615331 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.614946 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dv7\" (UniqueName: \"kubernetes.io/projected/e22030b7-c861-47ee-b7d7-75f6d659e9c2-kube-api-access-j2dv7\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.615331 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.615045 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-serving-cert\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.615558 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.615394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22030b7-c861-47ee-b7d7-75f6d659e9c2-tmp\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.615808 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.615783 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-config\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.615907 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.615821 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-oauth-serving-cert\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.615907 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.615821 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-service-ca\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.617492 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.617475 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-serving-cert\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.617537 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.617511 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-oauth-config\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.645211 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.645157 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dv7\" (UniqueName: \"kubernetes.io/projected/e22030b7-c861-47ee-b7d7-75f6d659e9c2-kube-api-access-j2dv7\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.686987 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.686954 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:08.899322 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.899233 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v998h" event={"ID":"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a","Type":"ContainerStarted","Data":"5bc7b0b7b31bf53d7846f8be2ee9ffd5caa807beed7aa0c61800119b74253beb"} Oct 30 16:21:08.900615 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.900569 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhm42" event={"ID":"c8e15255-91bb-411a-9c47-3c714735e70d","Type":"ContainerStarted","Data":"55e377cd2543dde7f785cffa2dbd8e0a59f376979a1b06d430400698f1e51f5b"} Oct 30 16:21:08.902226 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.902186 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qzkb" event={"ID":"d1dc93b8-6bd0-484e-a06c-d5a659e44eba","Type":"ContainerStarted","Data":"20f9ddd4923e8d214221d6e1ec5c49b35b4ce81f960838f4d6eab6694eaedf78"} Oct 30 16:21:08.902335 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.902238 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qzkb" event={"ID":"d1dc93b8-6bd0-484e-a06c-d5a659e44eba","Type":"ContainerStarted","Data":"07d57a9af5deb83e3e36e68a621dec198a98234087076fcea3f17542366a4eb5"} Oct 30 16:21:08.990973 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:08.989087 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-c966567fd-rv9sr"] Oct 30 16:21:08.994376 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:08.994351 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode22030b7_c861_47ee_b7d7_75f6d659e9c2.slice/crio-06f0c2cfa5ed9fff43f86ce77efaaab23ac240158b23d4a3f41d9713ea796c3c WatchSource:0}: Error finding container 06f0c2cfa5ed9fff43f86ce77efaaab23ac240158b23d4a3f41d9713ea796c3c: Status 404 returned error can't find the container with id 06f0c2cfa5ed9fff43f86ce77efaaab23ac240158b23d4a3f41d9713ea796c3c Oct 30 16:21:09.594257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.594171 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:09.594543 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.594520 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:09.594884 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.594870 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:09.617265 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.617232 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:21:09.617931 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.617298 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-46f47\"" Oct 30 16:21:09.617931 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.617372 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d8rcz\"" Oct 30 16:21:09.617931 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.617400 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:21:09.617931 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.617506 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:21:09.617931 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.617557 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m64dw\"" Oct 30 16:21:09.617931 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.617644 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:21:09.908777 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.908694 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c966567fd-rv9sr" event={"ID":"e22030b7-c861-47ee-b7d7-75f6d659e9c2","Type":"ContainerStarted","Data":"06f0c2cfa5ed9fff43f86ce77efaaab23ac240158b23d4a3f41d9713ea796c3c"} Oct 30 16:21:09.911628 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.911575 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qzkb" event={"ID":"d1dc93b8-6bd0-484e-a06c-d5a659e44eba","Type":"ContainerStarted","Data":"469ddd5b451948d283c7f0fa83b61c05d10e1d0a7d838fe127e0bce4c4460f55"} Oct 30 16:21:09.917467 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.917393 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57"] Oct 30 16:21:09.919826 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.919806 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:09.927458 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.927230 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:09.927458 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.927275 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:09.927458 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.927311 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:09.927458 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.927339 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmztd\" (UniqueName: \"kubernetes.io/projected/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-kube-api-access-qmztd\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:09.939927 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.939899 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vgrkr\"" Oct 30 16:21:09.942559 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.942536 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:21:09.942717 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.942621 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 16:21:09.943333 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.943224 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:21:09.943333 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.943280 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:21:09.943634 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.943542 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:21:09.946771 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.946748 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57"] Oct 30 16:21:09.950991 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.950968 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vgfj7"] Oct 30 16:21:09.954070 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.953195 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:09.954070 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.953755 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb"] Oct 30 16:21:09.957785 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.957721 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:09.977480 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.977455 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 16:21:09.977583 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.977462 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kx92s\"" Oct 30 16:21:09.977583 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.977464 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:21:09.977583 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.977504 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:21:09.977848 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.977655 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-95j9k\"" Oct 30 16:21:09.977848 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.977675 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:21:09.983805 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:09.983773 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb"] Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.028900 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cns2t\" (UniqueName: \"kubernetes.io/projected/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-kube-api-access-cns2t\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.028958 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029009 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029038 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029067 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029122 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a2b8eb86-b9d8-4e70-8c4e-736070d94081-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029149 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029177 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-tls\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029210 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmztd\" (UniqueName: \"kubernetes.io/projected/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-kube-api-access-qmztd\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029237 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.029257 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029265 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-sys\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.029937 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029287 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-metrics-client-ca\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.029937 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029314 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwlmr\" (UniqueName: \"kubernetes.io/projected/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-api-access-hwlmr\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.029937 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029759 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-root\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.029937 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029836 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:10.029937 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.029886 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.030235 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.030203 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:10.030345 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.030293 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-textfile\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.030398 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.030368 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:10.030644 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.030479 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-wtmp\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.034058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.034012 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:10.034797 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.034772 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:10.035767 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.035747 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:21:10.035869 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.035808 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 16:21:10.066166 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.066130 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmztd\" (UniqueName: \"kubernetes.io/projected/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-kube-api-access-qmztd\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:10.131615 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131550 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-wtmp\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.131615 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131619 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cns2t\" (UniqueName: \"kubernetes.io/projected/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-kube-api-access-cns2t\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.131863 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131663 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.131863 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131715 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.131863 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131746 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.131863 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131804 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a2b8eb86-b9d8-4e70-8c4e-736070d94081-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.131863 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131830 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.131863 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131858 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-tls\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.132117 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131892 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.132117 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131919 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-sys\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.132117 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131945 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-metrics-client-ca\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.132117 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.131971 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwlmr\" (UniqueName: \"kubernetes.io/projected/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-api-access-hwlmr\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.132117 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.132007 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-root\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.132117 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.132081 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.132117 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.132102 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-textfile\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.132456 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.132435 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-textfile\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.133459 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.132637 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-wtmp\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.133459 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.133145 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-metrics-client-ca\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.133459 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.133362 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-root\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.133860 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.133670 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.134218 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.134195 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.134312 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.134245 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.134312 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.134300 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-sys\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.134410 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.134383 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a2b8eb86-b9d8-4e70-8c4e-736070d94081-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.138121 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.138051 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.138121 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.138086 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-tls\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.138669 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.138627 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.140328 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.140290 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.160300 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.160238 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwlmr\" (UniqueName: \"kubernetes.io/projected/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-api-access-hwlmr\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.162527 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.162480 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cns2t\" (UniqueName: \"kubernetes.io/projected/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-kube-api-access-cns2t\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.236709 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.236674 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:10.277112 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.277069 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:10.285440 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.285412 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:10.313180 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:10.312849 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31a0ffa2_1bbf_4d1b_8a41_e0a8b709914a.slice/crio-5203e1370fb8a24e66347c1bd6dc51eed70d0103db315fa69856ca17b6ea0502 WatchSource:0}: Error finding container 5203e1370fb8a24e66347c1bd6dc51eed70d0103db315fa69856ca17b6ea0502: Status 404 returned error can't find the container with id 5203e1370fb8a24e66347c1bd6dc51eed70d0103db315fa69856ca17b6ea0502 Oct 30 16:21:10.567908 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.567875 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57"] Oct 30 16:21:10.571902 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:10.571859 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode01c7dab_d74b_4efe_9a0b_58a9eba6d956.slice/crio-ffc76cbf645fbace71ad36296e14f50e9dbbc00f6cc73378ee20857c7208ced7 WatchSource:0}: Error finding container ffc76cbf645fbace71ad36296e14f50e9dbbc00f6cc73378ee20857c7208ced7: Status 404 returned error can't find the container with id ffc76cbf645fbace71ad36296e14f50e9dbbc00f6cc73378ee20857c7208ced7 Oct 30 16:21:10.613029 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.613003 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb"] Oct 30 16:21:10.924562 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.924493 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" event={"ID":"a2b8eb86-b9d8-4e70-8c4e-736070d94081","Type":"ContainerStarted","Data":"dd3682e2ac5a533d0de34eb03be249a7edb22151f8465e37daa92fcc63ce6a76"} Oct 30 16:21:10.925918 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.925851 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgfj7" event={"ID":"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a","Type":"ContainerStarted","Data":"5203e1370fb8a24e66347c1bd6dc51eed70d0103db315fa69856ca17b6ea0502"} Oct 30 16:21:10.928628 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.928513 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" event={"ID":"e01c7dab-d74b-4efe-9a0b-58a9eba6d956","Type":"ContainerStarted","Data":"bee60c0fac4ecff0e1be39251a1a645b7aa66d045500204bfe64de9019ee4bc3"} Oct 30 16:21:10.928628 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.928550 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" event={"ID":"e01c7dab-d74b-4efe-9a0b-58a9eba6d956","Type":"ContainerStarted","Data":"52c853e8e522c8c91bae8422b7ac612839c96c17ccedbff728ac9c6b0bbcb4f5"} Oct 30 16:21:10.928628 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.928566 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" event={"ID":"e01c7dab-d74b-4efe-9a0b-58a9eba6d956","Type":"ContainerStarted","Data":"ffc76cbf645fbace71ad36296e14f50e9dbbc00f6cc73378ee20857c7208ced7"} Oct 30 16:21:10.930905 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:10.930821 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhm42" event={"ID":"c8e15255-91bb-411a-9c47-3c714735e70d","Type":"ContainerStarted","Data":"f72798d58c6800ec54b2db1afdd17f9b9f85975a05d5813c8728e572e5b3dbfb"} Oct 30 16:21:11.429695 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.429659 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:21:11.433100 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.433074 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.444828 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.444803 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-out\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.444946 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.444854 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.444946 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.444904 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.444946 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.444927 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.445108 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.444978 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.445108 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.445002 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-web-config\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.445108 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.445041 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.445108 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.445068 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.445108 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.445094 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.445331 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.445116 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.445331 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.445155 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-kube-api-access-9cbmh\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.445331 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.445189 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.445331 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.445215 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.454539 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.454510 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 16:21:11.454702 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.454564 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 16:21:11.454702 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.454636 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cgp6b\"" Oct 30 16:21:11.454702 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.454650 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 16:21:11.454702 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.454698 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 16:21:11.454915 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.454753 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 16:21:11.454915 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.454511 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 16:21:11.454915 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.454825 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 16:21:11.454915 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.454897 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 16:21:11.474765 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.474735 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:21:11.474960 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.474944 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 16:21:11.546386 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546309 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546526 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546385 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546526 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546417 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546526 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546470 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546526 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546497 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-web-config\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546774 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546536 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546774 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546561 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546774 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546585 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546774 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546628 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546774 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546667 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-kube-api-access-9cbmh\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546774 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546706 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546774 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546732 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.546774 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.546759 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-out\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.548902 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.547897 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.548902 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.548322 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.548902 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.548802 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.554817 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.554768 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.555386 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.555339 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.555899 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.555852 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.556832 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.556786 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-web-config\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.558708 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.558654 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.560447 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.560410 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.560527 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.560484 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.560769 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.560750 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-out\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.562286 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.562262 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.578070 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.578046 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-kube-api-access-9cbmh\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.756149 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.756113 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:11.939196 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.939124 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6qzkb" event={"ID":"d1dc93b8-6bd0-484e-a06c-d5a659e44eba","Type":"ContainerStarted","Data":"ad266f3b4968cbd813923b91e6c9a2c644afbd584cf1a4c6b7d1ea8ceec8f19b"} Oct 30 16:21:11.943080 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.942560 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgfj7" event={"ID":"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a","Type":"ContainerStarted","Data":"90a971174a26fbb50a33127807e1f192126f21e3f65b94a667246c7b94b3339a"} Oct 30 16:21:11.948057 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.948022 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qhm42" event={"ID":"c8e15255-91bb-411a-9c47-3c714735e70d","Type":"ContainerStarted","Data":"4d2c45f0f872184a0d8fb46e597e45bd31682a391ce6e990047146656785fc28"} Oct 30 16:21:11.948625 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.948549 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:11.986452 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:11.986389 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-6qzkb" podStartSLOduration=2.3247015859999998 podStartE2EDuration="4.986368549s" totalImagesPullingTime="2.661666963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 16:21:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:11.985551262 +0000 UTC m=+52.932933933" watchObservedRunningTime="2026-10-30 16:21:11.986368549 +0000 UTC m=+52.933751220" Oct 30 16:21:12.083749 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.083701 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-qhm42" podStartSLOduration=2.858211172 podStartE2EDuration="5.08368698s" totalImagesPullingTime="2.225475808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:12.082388964 +0000 UTC m=+53.029771632" watchObservedRunningTime="2026-10-30 16:21:12.08368698 +0000 UTC m=+53.031069646" Oct 30 16:21:12.096068 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.095459 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:21:12.112843 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:12.112812 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e33d75_acf8_44b3_a6e3_7599b04fa064.slice/crio-371dd94fe0177fcf78a797a514dd4cbfba3ccc8d00c215f2e90734fbd5df3ad9 WatchSource:0}: Error finding container 371dd94fe0177fcf78a797a514dd4cbfba3ccc8d00c215f2e90734fbd5df3ad9: Status 404 returned error can't find the container with id 371dd94fe0177fcf78a797a514dd4cbfba3ccc8d00c215f2e90734fbd5df3ad9 Oct 30 16:21:12.858141 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.858109 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:12.959209 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.959162 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" event={"ID":"a2b8eb86-b9d8-4e70-8c4e-736070d94081","Type":"ContainerStarted","Data":"1c70b5ec8b774674e1b0f12494e75dce5014dc69847d8c639a8f8a4e5a8e44d9"} Oct 30 16:21:12.959670 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.959287 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" event={"ID":"a2b8eb86-b9d8-4e70-8c4e-736070d94081","Type":"ContainerStarted","Data":"aa4c3662ed0cfb4c78c41b1c1f1a7bea6f7bd55edc718a7ed550544828e3d673"} Oct 30 16:21:12.959670 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.959330 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" event={"ID":"a2b8eb86-b9d8-4e70-8c4e-736070d94081","Type":"ContainerStarted","Data":"3eebcc0aba4509a4f380f953d3d5910e49e308d9cd82fb4f1ca49011098e3847"} Oct 30 16:21:12.962490 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.962461 2579 generic.go:356] "Generic (PLEG): container finished" podID="31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a" containerID="90a971174a26fbb50a33127807e1f192126f21e3f65b94a667246c7b94b3339a" exitCode=0 Oct 30 16:21:12.962632 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.962531 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgfj7" event={"ID":"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a","Type":"ContainerDied","Data":"90a971174a26fbb50a33127807e1f192126f21e3f65b94a667246c7b94b3339a"} Oct 30 16:21:12.966371 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.965909 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" event={"ID":"e01c7dab-d74b-4efe-9a0b-58a9eba6d956","Type":"ContainerStarted","Data":"5245c58b2e287c88ee1c7648e33b2c4953af4aef7e78ba59b48503dbf9f312be"} Oct 30 16:21:12.967786 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.967756 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-v998h" event={"ID":"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a","Type":"ContainerStarted","Data":"58acf02515312db8a6ce2467ed39d602756d7eee3cb061fc96aa01052f83d556"} Oct 30 16:21:12.969332 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:12.969300 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerStarted","Data":"371dd94fe0177fcf78a797a514dd4cbfba3ccc8d00c215f2e90734fbd5df3ad9"} Oct 30 16:21:13.009535 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:13.009474 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" podStartSLOduration=2.207439213 podStartE2EDuration="4.009454856s" totalImagesPullingTime="1.802015643s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:13.009157587 +0000 UTC m=+53.956540254" watchObservedRunningTime="2026-10-30 16:21:13.009454856 +0000 UTC m=+53.956837524" Oct 30 16:21:13.055520 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:13.055480 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-v998h" podStartSLOduration=2.169987614 podStartE2EDuration="6.05546784s" totalImagesPullingTime="3.885480226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:13.054457678 +0000 UTC m=+54.001840344" watchObservedRunningTime="2026-10-30 16:21:13.05546784 +0000 UTC m=+54.002850504" Oct 30 16:21:13.101899 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:13.101844 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" podStartSLOduration=2.745079722 podStartE2EDuration="4.101826941s" totalImagesPullingTime="1.356747219s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:13.101402114 +0000 UTC m=+54.048784781" watchObservedRunningTime="2026-10-30 16:21:13.101826941 +0000 UTC m=+54.049209610" Oct 30 16:21:13.973591 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:13.973561 2579 generic.go:356] "Generic (PLEG): container finished" podID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerID="a2e5530d27d52d0b4ff9b07d782b7281dc740e638b0f0d7d2e3c973d93ca9b26" exitCode=0 Oct 30 16:21:13.974099 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:13.973644 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerDied","Data":"a2e5530d27d52d0b4ff9b07d782b7281dc740e638b0f0d7d2e3c973d93ca9b26"} Oct 30 16:21:13.975561 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:13.975522 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c966567fd-rv9sr" event={"ID":"e22030b7-c861-47ee-b7d7-75f6d659e9c2","Type":"ContainerStarted","Data":"669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed"} Oct 30 16:21:13.977585 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:13.977560 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgfj7" event={"ID":"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a","Type":"ContainerStarted","Data":"a510247451b9ef8a07d96c9762ee6f54b4ce5470acb35382492ef035a2cea749"} Oct 30 16:21:13.977676 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:13.977626 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vgfj7" event={"ID":"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a","Type":"ContainerStarted","Data":"0642fed4281f498f922f7cedffcada4d0e8a3b56e28b04c508385ead7949999d"} Oct 30 16:21:14.090875 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.090826 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-c966567fd-rv9sr" podStartSLOduration=1.777284725 podStartE2EDuration="6.090807913s" totalImagesPullingTime="4.313523188s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:14.090309637 +0000 UTC m=+55.037692304" watchObservedRunningTime="2026-10-30 16:21:14.090807913 +0000 UTC m=+55.038190584" Oct 30 16:21:14.138401 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.138343 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vgfj7" podStartSLOduration=3.562775141 podStartE2EDuration="5.138329802s" totalImagesPullingTime="1.491164892s" totalInitContainerRuntime="84.389769ms" isStatefulPod=true podCreationTimestamp="2026-10-30 16:21:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:14.137439607 +0000 UTC m=+55.084822271" watchObservedRunningTime="2026-10-30 16:21:14.138329802 +0000 UTC m=+55.085712469" Oct 30 16:21:14.414622 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.414581 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-5f87899b87-ckzpf"] Oct 30 16:21:14.419489 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.419457 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.438889 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.438861 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 16:21:14.438889 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.438861 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 16:21:14.439127 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.438861 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 16:21:14.439127 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.438912 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ersnsplrsmf0e\"" Oct 30 16:21:14.439127 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.438914 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-r9gmj\"" Oct 30 16:21:14.439127 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.439042 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 16:21:14.440356 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.440334 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f87899b87-ckzpf"] Oct 30 16:21:14.500404 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.500371 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqrzv\" (UniqueName: \"kubernetes.io/projected/9f3cf821-229a-4690-abad-332006a3882a-kube-api-access-rqrzv\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.500561 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.500426 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-metrics-server-audit-profiles\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.500643 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.500586 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.500684 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.500669 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-tls\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.500720 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.500709 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-client-certs\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.500758 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.500734 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f3cf821-229a-4690-abad-332006a3882a-audit-log\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.500860 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.500838 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-client-ca-bundle\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.601803 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.601761 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-metrics-server-audit-profiles\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.601965 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.601935 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.601965 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.601962 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-tls\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.602071 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.601997 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-client-certs\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.602071 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.602022 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f3cf821-229a-4690-abad-332006a3882a-audit-log\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.602071 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.602058 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-client-ca-bundle\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.602221 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.602086 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrzv\" (UniqueName: \"kubernetes.io/projected/9f3cf821-229a-4690-abad-332006a3882a-kube-api-access-rqrzv\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.602581 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.602548 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f3cf821-229a-4690-abad-332006a3882a-audit-log\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.602726 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.602642 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.602927 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.602906 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-metrics-server-audit-profiles\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.604974 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.604947 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-client-ca-bundle\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.605081 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.605057 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-tls\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.605081 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.605069 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-client-certs\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.627653 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.627621 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrzv\" (UniqueName: \"kubernetes.io/projected/9f3cf821-229a-4690-abad-332006a3882a-kube-api-access-rqrzv\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.729906 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.729817 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:14.780318 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.778693 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr"] Oct 30 16:21:14.782205 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.782181 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:14.804826 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.802851 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 16:21:14.804826 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.803752 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-46k7l\"" Oct 30 16:21:14.804826 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.804759 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr"] Oct 30 16:21:14.905314 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.905280 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flf2g\" (UniqueName: \"kubernetes.io/projected/9b24f224-6dab-4478-890b-9da0d0679187-kube-api-access-flf2g\") pod \"monitoring-plugin-77f4f76594-hlrgr\" (UID: \"9b24f224-6dab-4478-890b-9da0d0679187\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:14.905474 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:14.905322 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b24f224-6dab-4478-890b-9da0d0679187-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hlrgr\" (UID: \"9b24f224-6dab-4478-890b-9da0d0679187\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:15.006387 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.006357 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flf2g\" (UniqueName: \"kubernetes.io/projected/9b24f224-6dab-4478-890b-9da0d0679187-kube-api-access-flf2g\") pod \"monitoring-plugin-77f4f76594-hlrgr\" (UID: \"9b24f224-6dab-4478-890b-9da0d0679187\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:15.006907 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.006401 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b24f224-6dab-4478-890b-9da0d0679187-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hlrgr\" (UID: \"9b24f224-6dab-4478-890b-9da0d0679187\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:15.009240 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.009215 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b24f224-6dab-4478-890b-9da0d0679187-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hlrgr\" (UID: \"9b24f224-6dab-4478-890b-9da0d0679187\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:15.032245 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.032222 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flf2g\" (UniqueName: \"kubernetes.io/projected/9b24f224-6dab-4478-890b-9da0d0679187-kube-api-access-flf2g\") pod \"monitoring-plugin-77f4f76594-hlrgr\" (UID: \"9b24f224-6dab-4478-890b-9da0d0679187\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:15.046735 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.046667 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-5f87899b87-ckzpf"] Oct 30 16:21:15.048034 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:15.048004 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3cf821_229a_4690_abad_332006a3882a.slice/crio-ca9666220a2ed5ac47c7b925b0a95bb4cf61ddd0852cd2c5aa9a93f01163b338 WatchSource:0}: Error finding container ca9666220a2ed5ac47c7b925b0a95bb4cf61ddd0852cd2c5aa9a93f01163b338: Status 404 returned error can't find the container with id ca9666220a2ed5ac47c7b925b0a95bb4cf61ddd0852cd2c5aa9a93f01163b338 Oct 30 16:21:15.104404 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.104371 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:15.402997 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.402955 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr"] Oct 30 16:21:15.404592 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:15.404554 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b24f224_6dab_4478_890b_9da0d0679187.slice/crio-6cdafef6533e60bb7f0c68a6000af6a1432653a56012191796e66690a7ee449d WatchSource:0}: Error finding container 6cdafef6533e60bb7f0c68a6000af6a1432653a56012191796e66690a7ee449d: Status 404 returned error can't find the container with id 6cdafef6533e60bb7f0c68a6000af6a1432653a56012191796e66690a7ee449d Oct 30 16:21:15.995213 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.995176 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerStarted","Data":"dcceb272812eb79c1bbd2ab10113ad8ea84053a65ab531aefe91c7b2a8977add"} Oct 30 16:21:15.995389 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.995267 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerStarted","Data":"16fdba6dda4782e29fc5b9866a5d9e996ff8737fb6b47d979212a7bb40e79415"} Oct 30 16:21:15.995389 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.995310 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerStarted","Data":"3b12bba989d729d6102b23b4ab112389882af4ea2af10d31b16ac16790b89ed9"} Oct 30 16:21:15.995389 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.995348 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerStarted","Data":"8b9e0a723a49c534b112e97c83f0933c27c9b33af7e05276efdd10acce8b300e"} Oct 30 16:21:15.995389 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.995387 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerStarted","Data":"52a15ac6f58e4511905538277767d83c9d89e40a552b1dcdcad33de6f837429f"} Oct 30 16:21:15.996664 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.996593 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" event={"ID":"9b24f224-6dab-4478-890b-9da0d0679187","Type":"ContainerStarted","Data":"6cdafef6533e60bb7f0c68a6000af6a1432653a56012191796e66690a7ee449d"} Oct 30 16:21:15.997912 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:15.997889 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" event={"ID":"9f3cf821-229a-4690-abad-332006a3882a","Type":"ContainerStarted","Data":"ca9666220a2ed5ac47c7b925b0a95bb4cf61ddd0852cd2c5aa9a93f01163b338"} Oct 30 16:21:16.996742 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:16.996708 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-d8cfd65cf-hkkx4"] Oct 30 16:21:16.998934 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:16.998916 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.008195 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.007510 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerStarted","Data":"287dd45d55fd2d5aea6a8ccb2844101819747e8229812cbb8afb1ddd7d3cdd87"} Oct 30 16:21:17.009853 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.009829 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" event={"ID":"9b24f224-6dab-4478-890b-9da0d0679187","Type":"ContainerStarted","Data":"28c7aa6587c8db2e0138b4976461b68e4f4bec7f98f29a332032d7ef20013111"} Oct 30 16:21:17.010023 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.010008 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:17.011486 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.011463 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" event={"ID":"9f3cf821-229a-4690-abad-332006a3882a","Type":"ContainerStarted","Data":"7e82300cc0cbc68003389af99a0c62bda8ad56881a5a5c642ea9bcd18fe20326"} Oct 30 16:21:17.014842 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.014821 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:17.024877 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.024859 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d8cfd65cf-hkkx4"] Oct 30 16:21:17.107865 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.107792 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 16:21:17.108006 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.107944 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" podStartSLOduration=1.611985932 podStartE2EDuration="3.107931954s" totalImagesPullingTime="1.495946022s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:17.107794853 +0000 UTC m=+58.055177524" watchObservedRunningTime="2026-10-30 16:21:17.107931954 +0000 UTC m=+58.055314618" Oct 30 16:21:17.129081 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.129053 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-service-ca\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.129236 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.129143 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-oauth-serving-cert\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.129703 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.129448 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kmlq\" (UniqueName: \"kubernetes.io/projected/b8014993-7131-499c-98b3-57b1b7429729-kube-api-access-5kmlq\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.129808 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.129757 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-trusted-ca-bundle\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.129958 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.129937 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-serving-cert\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.130085 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.129996 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-console-config\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.130085 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.130026 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8014993-7131-499c-98b3-57b1b7429729-tmp\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.130192 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.130095 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-oauth-config\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.152818 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.152769 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" podStartSLOduration=1.75322438 podStartE2EDuration="3.152758326s" totalImagesPullingTime="1.399533946s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:17.151677099 +0000 UTC m=+58.099059767" watchObservedRunningTime="2026-10-30 16:21:17.152758326 +0000 UTC m=+58.100140994" Oct 30 16:21:17.213162 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.213106 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.832874308 podStartE2EDuration="6.213091698s" totalImagesPullingTime="3.365567896s" totalInitContainerRuntime="14.649494ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:17.212171169 +0000 UTC m=+58.159553860" watchObservedRunningTime="2026-10-30 16:21:17.213091698 +0000 UTC m=+58.160474365" Oct 30 16:21:17.231136 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.231109 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-trusted-ca-bundle\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.231292 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.231151 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-serving-cert\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.231292 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.231169 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-console-config\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.231292 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.231284 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8014993-7131-499c-98b3-57b1b7429729-tmp\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.231447 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.231317 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-oauth-config\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.231447 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.231351 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-service-ca\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.231447 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.231412 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-oauth-serving-cert\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.231586 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.231491 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kmlq\" (UniqueName: \"kubernetes.io/projected/b8014993-7131-499c-98b3-57b1b7429729-kube-api-access-5kmlq\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.231940 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.231908 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8014993-7131-499c-98b3-57b1b7429729-tmp\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.232058 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.231945 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-console-config\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.232224 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.232204 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-service-ca\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.232299 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.232252 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-trusted-ca-bundle\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.232299 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.232279 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-oauth-serving-cert\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.234418 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.234394 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-oauth-config\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.234500 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.234487 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-serving-cert\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.256503 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.256483 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kmlq\" (UniqueName: \"kubernetes.io/projected/b8014993-7131-499c-98b3-57b1b7429729-kube-api-access-5kmlq\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.310499 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.310468 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:17.606091 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:17.604360 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-d8cfd65cf-hkkx4"] Oct 30 16:21:18.016825 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:18.016774 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8cfd65cf-hkkx4" event={"ID":"b8014993-7131-499c-98b3-57b1b7429729","Type":"ContainerStarted","Data":"011ef189e36172aeb318c7ab823e71b80b8dce90ce6d3b544ffb956605000904"} Oct 30 16:21:18.017326 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:18.016846 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8cfd65cf-hkkx4" event={"ID":"b8014993-7131-499c-98b3-57b1b7429729","Type":"ContainerStarted","Data":"a74ad2aa4355d3eeb56a53a285b0325202d10f7296bbf4436e85a22a4a76007e"} Oct 30 16:21:18.064253 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:18.064202 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-d8cfd65cf-hkkx4" podStartSLOduration=2.064188076 podStartE2EDuration="2.064188076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:18.063209571 +0000 UTC m=+59.010592238" watchObservedRunningTime="2026-10-30 16:21:18.064188076 +0000 UTC m=+59.011570789" Oct 30 16:21:18.687773 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:18.687740 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:18.687920 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:18.687848 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:18.692520 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:18.692502 2579 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:19.023952 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:19.023926 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:22.251559 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.251528 2579 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d8cfd65cf-hkkx4"] Oct 30 16:21:22.289773 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.289737 2579 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-565777cb7f-d862k"] Oct 30 16:21:22.293193 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.293173 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.317043 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.317012 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565777cb7f-d862k"] Oct 30 16:21:22.378255 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.378218 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-tmp\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.378419 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.378290 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6gbn\" (UniqueName: \"kubernetes.io/projected/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-kube-api-access-z6gbn\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.378461 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.378424 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-config\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.378499 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.378473 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-serving-cert\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.378568 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.378551 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-trusted-ca-bundle\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.378617 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.378578 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-oauth-config\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.378654 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.378646 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-oauth-serving-cert\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.378702 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.378688 2579 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-service-ca\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.479712 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.479679 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-serving-cert\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.479908 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.479725 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-trusted-ca-bundle\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.479908 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.479854 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-oauth-config\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.479908 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.479904 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-oauth-serving-cert\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.480080 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.479972 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-service-ca\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.480080 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.480016 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-tmp\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.480080 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.480070 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6gbn\" (UniqueName: \"kubernetes.io/projected/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-kube-api-access-z6gbn\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.480218 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.480119 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-config\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.480550 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.480529 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-oauth-serving-cert\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.480736 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.480710 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-service-ca\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.480813 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.480739 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-trusted-ca-bundle\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.480909 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.480891 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-config\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.480951 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.480928 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-tmp\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.483011 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.482983 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-serving-cert\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.483160 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.483144 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-oauth-config\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.506105 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.506022 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6gbn\" (UniqueName: \"kubernetes.io/projected/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-kube-api-access-z6gbn\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.604509 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.604474 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:22.913164 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.913138 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-565777cb7f-d862k"] Oct 30 16:21:22.915086 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:22.915060 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53ecbbc_c6c6_40ed_9054_94e5d25a8ea8.slice/crio-32da6cefad5f9764cf3178c7c6878e4ded6d15d81574464735d04eb0da6a5703 WatchSource:0}: Error finding container 32da6cefad5f9764cf3178c7c6878e4ded6d15d81574464735d04eb0da6a5703: Status 404 returned error can't find the container with id 32da6cefad5f9764cf3178c7c6878e4ded6d15d81574464735d04eb0da6a5703 Oct 30 16:21:22.982671 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:22.982648 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:23.038473 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:23.038436 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565777cb7f-d862k" event={"ID":"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8","Type":"ContainerStarted","Data":"0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4"} Oct 30 16:21:23.038680 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:23.038494 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565777cb7f-d862k" event={"ID":"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8","Type":"ContainerStarted","Data":"32da6cefad5f9764cf3178c7c6878e4ded6d15d81574464735d04eb0da6a5703"} Oct 30 16:21:23.089148 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:23.089094 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-565777cb7f-d862k" podStartSLOduration=1.089079022 podStartE2EDuration="1.089079022s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:23.08781042 +0000 UTC m=+64.035193086" watchObservedRunningTime="2026-10-30 16:21:23.089079022 +0000 UTC m=+64.036461688" Oct 30 16:21:25.323030 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.322994 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:25.323030 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.323039 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:25.353716 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.353685 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:21:25.353716 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.353685 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:21:25.366154 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.366125 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:25.366290 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.366164 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:25.423831 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.423793 2579 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:25.439361 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.439332 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:21:25.459819 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.459794 2579 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:21:25.467397 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.467366 2579 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:25.530849 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.530820 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m64dw\"" Oct 30 16:21:25.535942 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.535924 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:25.544007 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.543986 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d8rcz\"" Oct 30 16:21:25.549269 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.549243 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:25.551621 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.551570 2579 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-46f47\"" Oct 30 16:21:25.556973 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.556944 2579 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:25.838843 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.838819 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-gccnz"] Oct 30 16:21:25.840350 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:25.840318 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67d3a22c_930e_4113_ba61_2aecccabf9d2.slice/crio-dad6a8a9012b28ad43afa0d0739af1dc300a4ad8ba17f721a039d42d8e0a3e28 WatchSource:0}: Error finding container dad6a8a9012b28ad43afa0d0739af1dc300a4ad8ba17f721a039d42d8e0a3e28: Status 404 returned error can't find the container with id dad6a8a9012b28ad43afa0d0739af1dc300a4ad8ba17f721a039d42d8e0a3e28 Oct 30 16:21:25.857802 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.857776 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-9ls8t"] Oct 30 16:21:25.858252 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:25.858233 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569992a0_aa75_48cb_ad8d_fb61295a5093.slice/crio-5b9c936fbf7990544e33c048a35d79575c507dbc22ae6732a20e8c5bff813be1 WatchSource:0}: Error finding container 5b9c936fbf7990544e33c048a35d79575c507dbc22ae6732a20e8c5bff813be1: Status 404 returned error can't find the container with id 5b9c936fbf7990544e33c048a35d79575c507dbc22ae6732a20e8c5bff813be1 Oct 30 16:21:25.879881 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:25.879858 2579 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-26nhz"] Oct 30 16:21:25.880111 ip-10-0-128-181 kubenswrapper[2579]: W1030 16:21:25.880093 2579 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda3c94aa_c880_4844_8cc8_2182ff86e321.slice/crio-dc70e5caa9a93c9ec992515edfba0352a26ee50ac8667eecbcfabb922185ac8e WatchSource:0}: Error finding container dc70e5caa9a93c9ec992515edfba0352a26ee50ac8667eecbcfabb922185ac8e: Status 404 returned error can't find the container with id dc70e5caa9a93c9ec992515edfba0352a26ee50ac8667eecbcfabb922185ac8e Oct 30 16:21:26.050537 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:26.050500 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-26nhz" event={"ID":"da3c94aa-c880-4844-8cc8-2182ff86e321","Type":"ContainerStarted","Data":"dc70e5caa9a93c9ec992515edfba0352a26ee50ac8667eecbcfabb922185ac8e"} Oct 30 16:21:26.051655 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:26.051629 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gccnz" event={"ID":"67d3a22c-930e-4113-ba61-2aecccabf9d2","Type":"ContainerStarted","Data":"dad6a8a9012b28ad43afa0d0739af1dc300a4ad8ba17f721a039d42d8e0a3e28"} Oct 30 16:21:26.052697 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:26.052675 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9ls8t" event={"ID":"569992a0-aa75-48cb-ad8d-fb61295a5093","Type":"ContainerStarted","Data":"5b9c936fbf7990544e33c048a35d79575c507dbc22ae6732a20e8c5bff813be1"} Oct 30 16:21:27.059877 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:27.059848 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9ls8t" event={"ID":"569992a0-aa75-48cb-ad8d-fb61295a5093","Type":"ContainerStarted","Data":"0d66cbe9620c4cb333520e637cc1d16b5456011707cd23a25f2902242b083e16"} Oct 30 16:21:27.310902 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:27.310866 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:28.067175 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:28.067135 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-9ls8t" event={"ID":"569992a0-aa75-48cb-ad8d-fb61295a5093","Type":"ContainerStarted","Data":"fd7f36e016cdf45e8460eeb5b505c6ee6f327d343565e2a731119c1db04ddc61"} Oct 30 16:21:29.073370 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:29.073331 2579 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-gccnz" event={"ID":"67d3a22c-930e-4113-ba61-2aecccabf9d2","Type":"ContainerStarted","Data":"1ae1de8e2e02ded9058002c494ec1803733ba8f6add4a0de303a9fe088800967"} Oct 30 16:21:29.117842 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:29.117794 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-gccnz" podStartSLOduration=66.976471347 podStartE2EDuration="1m10.117778507s" totalImagesPullingTime="3.14130716s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:29.116823223 +0000 UTC m=+70.064205891" watchObservedRunningTime="2026-10-30 16:21:29.117778507 +0000 UTC m=+70.065161174" Oct 30 16:21:29.118005 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:29.117881 2579 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-9ls8t" podStartSLOduration=69.102631964 podStartE2EDuration="1m10.117874303s" totalImagesPullingTime="1.015242339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:20:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:28.112440665 +0000 UTC m=+69.059823331" watchObservedRunningTime="2026-10-30 16:21:29.117874303 +0000 UTC m=+70.065256971" Oct 30 16:21:30.077198 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:30.077159 2579 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:30.790297 ip-10-0-128-181 kubenswrapper[2579]: I1030 16:21:30.790151 2579 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:21:30.790299 ip-10-0-128-181 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 16:21:30.801966 ip-10-0-128-181 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 16:21:30.802130 ip-10-0-128-181 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 16:21:30.802178 ip-10-0-128-181 systemd[1]: kubelet.service: Consumed 4.069s CPU time. Oct 30 16:21:30.817424 ip-10-0-128-181 systemd[1]: Starting Kubernetes Kubelet... Oct 30 16:21:30.879096 ip-10-0-128-181 kubenswrapper[6400]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:21:30.879096 ip-10-0-128-181 kubenswrapper[6400]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 16:21:30.879096 ip-10-0-128-181 kubenswrapper[6400]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:21:30.879096 ip-10-0-128-181 kubenswrapper[6400]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 16:21:30.881162 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881083 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:30.881162 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881162 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:30.881162 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881166 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881169 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881172 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881175 6400 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881178 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881180 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881183 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881186 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881188 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881191 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881193 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881195 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881198 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881200 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881202 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881205 6400 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881207 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881210 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881213 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881215 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:30.881253 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881224 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881227 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881233 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881238 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881242 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881246 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881249 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881252 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881255 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881258 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881260 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881264 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881266 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881269 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881272 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881274 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881277 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881279 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881282 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:30.881778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881284 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881287 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881289 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881292 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881294 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881297 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881299 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881301 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881304 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881306 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881309 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881312 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881316 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881319 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881322 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881324 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881327 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881329 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881332 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881334 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:30.882221 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881337 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881339 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881342 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881344 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881347 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881350 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881352 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881354 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881357 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881359 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881362 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881366 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881368 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881371 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881374 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881376 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881379 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881382 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881385 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:30.882746 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881388 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881390 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881393 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881395 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881398 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881400 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881403 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881406 6400 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881408 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881411 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881414 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881416 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881420 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881422 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881425 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881427 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881429 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881432 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881434 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881436 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:30.883195 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881439 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881444 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881447 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881449 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881452 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.881454 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882199 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882205 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882208 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882211 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882214 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882216 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882219 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882221 6400 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882224 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882227 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882230 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882232 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882235 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:30.883717 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882238 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882241 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882243 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882246 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882249 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882251 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882254 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882256 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882259 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882261 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882264 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882267 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882269 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882272 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882275 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882278 6400 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882280 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882283 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882286 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:30.884218 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882288 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882290 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882293 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882296 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882299 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882301 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882304 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882306 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882308 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882311 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882313 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882316 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882320 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882324 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882328 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882331 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882334 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882336 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882339 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:30.884669 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882341 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882344 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882347 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882350 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882352 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882355 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882357 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882360 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882362 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882365 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882367 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882370 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882372 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882374 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882377 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882381 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882384 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882387 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882390 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882392 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:30.885126 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882395 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882398 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882400 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882402 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882405 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882408 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882410 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882412 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882415 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882418 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882420 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882423 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882425 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882428 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882430 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882433 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882435 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882438 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882440 6400 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:30.885594 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882443 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882445 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882448 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882450 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882453 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882456 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882458 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882461 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882464 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882467 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882469 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882471 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882474 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882476 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882479 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882481 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.882993 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883000 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883003 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883005 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:30.886073 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883008 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883011 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883014 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883016 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883019 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883021 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883024 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883029 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883032 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883035 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883037 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883040 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883042 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883045 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883047 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883050 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883052 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883055 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883059 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:30.886538 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883063 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883066 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883070 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883073 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883076 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883079 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883082 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883084 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883087 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883090 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883092 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883095 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883097 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883099 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883102 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883110 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883113 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883116 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883118 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:30.887003 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883120 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883125 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883127 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883130 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883132 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883135 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883137 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883140 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883142 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883145 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883147 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883150 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883152 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883155 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883157 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883160 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883162 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883165 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883168 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:30.887499 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883170 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883172 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883176 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883178 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883181 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883183 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883186 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883188 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883190 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883193 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883201 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883204 6400 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883206 6400 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883209 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883213 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883215 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883218 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883220 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883223 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883225 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883227 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:30.888198 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883230 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883232 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883235 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883237 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883240 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883242 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883247 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883251 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883254 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883256 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883259 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883262 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883264 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883266 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883269 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883272 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883274 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883276 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883279 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:30.888898 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883281 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883284 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883286 6400 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883300 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.883304 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883396 6400 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883404 6400 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883415 6400 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883419 6400 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883424 6400 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883427 6400 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883431 6400 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883435 6400 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883438 6400 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883441 6400 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883445 6400 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883448 6400 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883450 6400 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883453 6400 flags.go:64] FLAG: --cgroup-root="" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883458 6400 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883461 6400 flags.go:64] FLAG: --client-ca-file="" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883464 6400 flags.go:64] FLAG: --cloud-provider="external" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883467 6400 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883473 6400 flags.go:64] FLAG: --cluster-domain="" Oct 30 16:21:30.889348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883476 6400 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883479 6400 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883482 6400 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883485 6400 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883489 6400 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883492 6400 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883495 6400 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883498 6400 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883501 6400 flags.go:64] FLAG: --contention-profiling="false" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883504 6400 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883507 6400 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883510 6400 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883518 6400 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883523 6400 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883527 6400 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883530 6400 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883533 6400 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883536 6400 flags.go:64] FLAG: --enable-server="true" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883539 6400 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883546 6400 flags.go:64] FLAG: --event-burst="100" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883549 6400 flags.go:64] FLAG: --event-qps="50" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883552 6400 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883555 6400 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883558 6400 flags.go:64] FLAG: --eviction-hard="" Oct 30 16:21:30.889964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883562 6400 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883565 6400 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883568 6400 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883572 6400 flags.go:64] FLAG: --eviction-soft="" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883575 6400 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883578 6400 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883581 6400 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883583 6400 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883586 6400 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883590 6400 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883592 6400 flags.go:64] FLAG: --feature-gates="" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883596 6400 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883613 6400 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883616 6400 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883620 6400 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883623 6400 flags.go:64] FLAG: --healthz-port="10248" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883625 6400 flags.go:64] FLAG: --help="false" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883628 6400 flags.go:64] FLAG: --hostname-override="ip-10-0-128-181.ec2.internal" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883631 6400 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883634 6400 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883637 6400 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883646 6400 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883651 6400 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883654 6400 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 16:21:30.890530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883657 6400 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883660 6400 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883663 6400 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883666 6400 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883669 6400 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883672 6400 flags.go:64] FLAG: --kube-reserved="" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883675 6400 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883678 6400 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883681 6400 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883684 6400 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883687 6400 flags.go:64] FLAG: --lock-file="" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883691 6400 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883694 6400 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883697 6400 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883703 6400 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883706 6400 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883709 6400 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883711 6400 flags.go:64] FLAG: --logging-format="text" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883714 6400 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883718 6400 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883721 6400 flags.go:64] FLAG: --manifest-url="" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883723 6400 flags.go:64] FLAG: --manifest-url-header="" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883728 6400 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883731 6400 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883735 6400 flags.go:64] FLAG: --max-pods="110" Oct 30 16:21:30.891122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883738 6400 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883741 6400 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883744 6400 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883747 6400 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883750 6400 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883760 6400 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883763 6400 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883769 6400 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883772 6400 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883775 6400 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883778 6400 flags.go:64] FLAG: --pod-cidr="" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883781 6400 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883784 6400 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883787 6400 flags.go:64] FLAG: --pods-per-core="0" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883790 6400 flags.go:64] FLAG: --port="10250" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883793 6400 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883795 6400 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-051aebc9d776fd5d0" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883799 6400 flags.go:64] FLAG: --qos-reserved="" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883803 6400 flags.go:64] FLAG: --read-only-port="10255" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883806 6400 flags.go:64] FLAG: --register-node="true" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883809 6400 flags.go:64] FLAG: --register-with-taints="" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883812 6400 flags.go:64] FLAG: --registry-burst="10" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883815 6400 flags.go:64] FLAG: --registry-qps="5" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883818 6400 flags.go:64] FLAG: --reserved-cpus="" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883821 6400 flags.go:64] FLAG: --reserved-memory="" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883825 6400 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 16:21:30.891743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883828 6400 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883831 6400 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883834 6400 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883837 6400 flags.go:64] FLAG: --runonce="false" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883840 6400 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883843 6400 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883846 6400 flags.go:64] FLAG: --seccomp-default="false" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883849 6400 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883852 6400 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883856 6400 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883859 6400 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883863 6400 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883871 6400 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883875 6400 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883877 6400 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883880 6400 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883883 6400 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883886 6400 flags.go:64] FLAG: --system-cgroups="" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883889 6400 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883894 6400 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883897 6400 flags.go:64] FLAG: --tls-cert-file="" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883901 6400 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883907 6400 flags.go:64] FLAG: --tls-min-version="" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883910 6400 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883914 6400 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 16:21:30.892370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883917 6400 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883919 6400 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883922 6400 flags.go:64] FLAG: --v="2" Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883927 6400 flags.go:64] FLAG: --version="false" Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883931 6400 flags.go:64] FLAG: --vmodule="" Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883935 6400 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.883938 6400 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884110 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884113 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884116 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884119 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884122 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884124 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884127 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884129 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884132 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884134 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884137 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884141 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884143 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884146 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:30.893100 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884149 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884152 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884154 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884157 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884159 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884162 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884164 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884167 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884169 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884172 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884176 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884178 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884182 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884187 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884190 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884193 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884195 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884198 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884200 6400 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:30.893596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884203 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884205 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884208 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884210 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884213 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884215 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884218 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884221 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884223 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884226 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884230 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884233 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884236 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884239 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884241 6400 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884244 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884246 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884249 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884251 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884253 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:30.894214 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884256 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884258 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884261 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884265 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884267 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884269 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884272 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884275 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884277 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884280 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884283 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884285 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884288 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884290 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884292 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884295 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884297 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884300 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884302 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884305 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884307 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:30.894759 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884309 6400 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884316 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884318 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884321 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884324 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884328 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884330 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884333 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884335 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884338 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884340 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884343 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884345 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884347 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884351 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884354 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884357 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884359 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884362 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:30.895361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884364 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884368 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884372 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884375 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884378 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884381 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884384 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884386 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884389 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884392 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884394 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884396 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884399 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.884425 6400 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884459 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:30.895906 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884465 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884468 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884471 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884474 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884477 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884480 6400 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884483 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884485 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884487 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884490 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884492 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884495 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884499 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884502 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884505 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884507 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884510 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884512 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884515 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884517 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:30.896321 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884520 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884523 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884525 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884527 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884530 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884532 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884535 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884537 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884539 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884542 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884545 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884547 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884551 6400 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884553 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884555 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884558 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884560 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884929 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884933 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884935 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:30.896886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884938 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884941 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884943 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884946 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884949 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884952 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884954 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884956 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884959 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884961 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884964 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884966 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884970 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884974 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884977 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884981 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884984 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884986 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884989 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:30.897405 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884991 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884994 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884997 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.884999 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885002 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885006 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885008 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885011 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885013 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885016 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885018 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885021 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885024 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885027 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885029 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885032 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885034 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885037 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885039 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:30.897938 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885042 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885044 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885046 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885049 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885051 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885054 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885056 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885058 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885061 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885063 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885065 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885068 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885071 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885075 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885078 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885081 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885083 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885086 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885089 6400 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:30.898388 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885092 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885095 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885097 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885100 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885102 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885105 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885109 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.885111 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.889828 6400 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.889844 6400 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.889865 6400 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.889870 6400 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889930 6400 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889935 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889940 6400 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889943 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889946 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889948 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889951 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889954 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:30.898944 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889957 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889960 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889963 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889965 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889968 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889971 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889974 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889977 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889980 6400 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889982 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889985 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889988 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889991 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889993 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889996 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.889999 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890001 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890004 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890007 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890009 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:30.899433 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890012 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890014 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890017 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890020 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890022 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890025 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890027 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890030 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890032 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890035 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890038 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890040 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890043 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890046 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890049 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890051 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890054 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890057 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890060 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890063 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:30.899922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890066 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890068 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890071 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890073 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890076 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890078 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890081 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890083 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890086 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890088 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890091 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890093 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890096 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890099 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890102 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890104 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890107 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890109 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890112 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:30.900395 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890115 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890117 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890120 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890122 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890124 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890127 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890129 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890132 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890135 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890137 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890141 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890145 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890150 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890153 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890156 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890159 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890161 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890164 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890167 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:30.900914 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890170 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890172 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890175 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890179 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890183 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890186 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890189 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890191 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890194 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890197 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890199 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890202 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890204 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890207 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890209 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890212 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890214 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890217 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890219 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:30.901361 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890222 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.890250 6400 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890397 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890402 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890405 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890409 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890412 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890416 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890419 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890421 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890424 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890426 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890429 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890432 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890434 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890437 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:30.901829 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890439 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890442 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890444 6400 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890447 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890449 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890452 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890455 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890457 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890460 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890462 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890465 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890468 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890470 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890473 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890475 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890478 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890480 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890483 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890485 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890488 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:30.902211 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890490 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890493 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890496 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890498 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890502 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890505 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890509 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890512 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890515 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890517 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890520 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890522 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890525 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890528 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890530 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890533 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890536 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890538 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890541 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:30.902787 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890544 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890546 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890549 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890551 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890554 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890556 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890559 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890561 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890564 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890566 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890569 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890572 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890574 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890577 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890579 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890582 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890584 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890586 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890589 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:30.903520 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890592 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890594 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890597 6400 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890619 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890622 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890626 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890631 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890634 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890639 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890641 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890644 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890646 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890649 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890652 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890655 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890657 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890660 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890663 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890665 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890668 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:30.904219 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890670 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890674 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890678 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890681 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890684 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890687 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890689 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890692 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890694 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890697 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890699 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890702 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890705 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890708 6400 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.890733 6400 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 16:21:30.904722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890764 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890768 6400 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890771 6400 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890774 6400 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890777 6400 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890780 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890783 6400 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890785 6400 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890788 6400 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890790 6400 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890793 6400 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890795 6400 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890798 6400 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890801 6400 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890804 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890806 6400 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890808 6400 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890811 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890813 6400 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 16:21:30.905179 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890816 6400 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890821 6400 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890824 6400 feature_gate.go:386] unrecognized feature gate: Example Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890827 6400 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890829 6400 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890832 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890834 6400 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890837 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890839 6400 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890842 6400 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890844 6400 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890847 6400 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890850 6400 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890852 6400 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890855 6400 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890857 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890860 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890862 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890865 6400 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890868 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 16:21:30.905663 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890870 6400 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890874 6400 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890878 6400 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890881 6400 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890884 6400 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890886 6400 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890889 6400 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890891 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890894 6400 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890896 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890899 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890901 6400 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890904 6400 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890907 6400 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890910 6400 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890912 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890914 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890917 6400 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890919 6400 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 16:21:30.906149 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890921 6400 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890924 6400 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890926 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890929 6400 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890931 6400 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890935 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890937 6400 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890939 6400 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890942 6400 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890944 6400 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890947 6400 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890950 6400 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890952 6400 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890955 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890957 6400 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890960 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890962 6400 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890965 6400 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890967 6400 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890969 6400 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 16:21:30.906679 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890972 6400 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890974 6400 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890976 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890979 6400 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890981 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890984 6400 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890987 6400 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890989 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890992 6400 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890994 6400 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890997 6400 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.890999 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891002 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891004 6400 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891007 6400 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891010 6400 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891014 6400 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891017 6400 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891020 6400 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 16:21:30.907156 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891023 6400 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891026 6400 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891028 6400 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891032 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891035 6400 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891037 6400 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891040 6400 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891042 6400 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:30.891045 6400 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.891105 6400 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.892061 6400 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.892118 6400 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.892291 6400 server.go:1014] "Starting client certificate rotation" Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.892440 6400 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.892503 6400 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 16:15:19 +0000 UTC" deadline="2028-05-11 14:41:01.365434163 +0000 UTC" Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.892543 6400 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13414h19m30.4728952s" Oct 30 16:21:30.907621 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.892768 6400 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:21:30.908052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.893816 6400 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 16:21:30.908052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.894939 6400 log.go:25] "Validated CRI v1 runtime API" Oct 30 16:21:30.908052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.896699 6400 log.go:25] "Validated CRI v1 image API" Oct 30 16:21:30.908052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.897420 6400 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 16:21:30.908052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.899536 6400 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 a2017790-fd04-4ac0-81c7-2185d1e2f434:/dev/nvme0n1p3 dbdda714-2453-4b06-b787-c206bcd7dfe0:/dev/nvme0n1p4] Oct 30 16:21:30.908203 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.899556 6400 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/06f0c2cfa5ed9fff43f86ce77efaaab23ac240158b23d4a3f41d9713ea796c3c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/06f0c2cfa5ed9fff43f86ce77efaaab23ac240158b23d4a3f41d9713ea796c3c/userdata/shm major:0 minor:226 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/07d57a9af5deb83e3e36e68a621dec198a98234087076fcea3f17542366a4eb5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/07d57a9af5deb83e3e36e68a621dec198a98234087076fcea3f17542366a4eb5/userdata/shm major:0 minor:203 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/09c1ac4d7705d3057cc77a8993d87c622e4d54efb2e60124b7819a6fa0f53fe3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/09c1ac4d7705d3057cc77a8993d87c622e4d54efb2e60124b7819a6fa0f53fe3/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/32da6cefad5f9764cf3178c7c6878e4ded6d15d81574464735d04eb0da6a5703/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/32da6cefad5f9764cf3178c7c6878e4ded6d15d81574464735d04eb0da6a5703/userdata/shm major:0 minor:413 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/371dd94fe0177fcf78a797a514dd4cbfba3ccc8d00c215f2e90734fbd5df3ad9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/371dd94fe0177fcf78a797a514dd4cbfba3ccc8d00c215f2e90734fbd5df3ad9/userdata/shm major:0 minor:293 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5203e1370fb8a24e66347c1bd6dc51eed70d0103db315fa69856ca17b6ea0502/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5203e1370fb8a24e66347c1bd6dc51eed70d0103db315fa69856ca17b6ea0502/userdata/shm major:0 minor:246 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/55e377cd2543dde7f785cffa2dbd8e0a59f376979a1b06d430400698f1e51f5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/55e377cd2543dde7f785cffa2dbd8e0a59f376979a1b06d430400698f1e51f5b/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5b9c936fbf7990544e33c048a35d79575c507dbc22ae6732a20e8c5bff813be1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5b9c936fbf7990544e33c048a35d79575c507dbc22ae6732a20e8c5bff813be1/userdata/shm major:0 minor:428 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5bc7b0b7b31bf53d7846f8be2ee9ffd5caa807beed7aa0c61800119b74253beb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5bc7b0b7b31bf53d7846f8be2ee9ffd5caa807beed7aa0c61800119b74253beb/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6cdafef6533e60bb7f0c68a6000af6a1432653a56012191796e66690a7ee449d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6cdafef6533e60bb7f0c68a6000af6a1432653a56012191796e66690a7ee449d/userdata/shm major:0 minor:350 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6d0ffdfc11a08f5cc3957740a7e3ef60833c74db082289fa4e381f10835cc270/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6d0ffdfc11a08f5cc3957740a7e3ef60833c74db082289fa4e381f10835cc270/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8c8175c5ed0e3f758f1224bb024bd1c027e3efdc79d3a8f2b8d4516b86fcc475/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8c8175c5ed0e3f758f1224bb024bd1c027e3efdc79d3a8f2b8d4516b86fcc475/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a42f88609e94dc9ee337ffff0f37772fbaef9f59f4cdf4df369a135c2a1bba4c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a42f88609e94dc9ee337ffff0f37772fbaef9f59f4cdf4df369a135c2a1bba4c/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a74ad2aa4355d3eeb56a53a285b0325202d10f7296bbf4436e85a22a4a76007e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a74ad2aa4355d3eeb56a53a285b0325202d10f7296bbf4436e85a22a4a76007e/userdata/shm major:0 minor:400 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a869f354c2abbc63e2aba5d9a42b0388132aa0f561bbfd6477624e707347ae4b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a869f354c2abbc63e2aba5d9a42b0388132aa0f561bbfd6477624e707347ae4b/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c480f118ec53bbba0e7858f55baeb2a3f1cf76b4ded3fd85b6d150ff40522aaa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c480f118ec53bbba0e7858f55baeb2a3f1cf76b4ded3fd85b6d150ff40522aaa/userdata/shm major:0 minor:70 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ca9666220a2ed5ac47c7b925b0a95bb4cf61ddd0852cd2c5aa9a93f01163b338/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ca9666220a2ed5ac47c7b925b0a95bb4cf61ddd0852cd2c5aa9a93f01163b338/userdata/shm major:0 minor:340 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0af2178deb53c5d43e8b86ff93686a2b479c30923bb0969a9ca457653639e35/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0af2178deb53c5d43e8b86ff93686a2b479c30923bb0969a9ca457653639e35/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8dd102e070f9a4d8557721bedc77be7ee8a47857446863ee6f9aea82af9ecdc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8dd102e070f9a4d8557721bedc77be7ee8a47857446863ee6f9aea82af9ecdc/userdata/shm major:0 minor:83 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dad6a8a9012b28ad43afa0d0739af1dc300a4ad8ba17f721a039d42d8e0a3e28/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dad6a8a9012b28ad43afa0d0739af1dc300a4ad8ba17f721a039d42d8e0a3e28/userdata/shm major:0 minor:426 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dc70e5caa9a93c9ec992515edfba0352a26ee50ac8667eecbcfabb922185ac8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dc70e5caa9a93c9ec992515edfba0352a26ee50ac8667eecbcfabb922185ac8e/userdata/shm major:0 minor:430 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/dd3682e2ac5a533d0de34eb03be249a7edb22151f8465e37daa92fcc63ce6a76/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/dd3682e2ac5a533d0de34eb03be249a7edb22151f8465e37daa92fcc63ce6a76/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e068c96992dad3f8ef6035a37c77dd7ddf57157a437c388c4a5573b1bd53a9a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e068c96992dad3f8ef6035a37c77dd7ddf57157a437c388c4a5573b1bd53a9a9/userdata/shm major:0 minor:74 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f696498181b3360832b434fed9f73d14e4a751195dcd8d9c3c64e7dd57ce6ffc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f696498181b3360832b434fed9f73d14e4a751195dcd8d9c3c64e7dd57ce6ffc/userdata/shm major:0 minor:100 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffc76cbf645fbace71ad36296e14f50e9dbbc00f6cc73378ee20857c7208ced7/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffc76cbf645fbace71ad36296e14f50e9dbbc00f6cc73378ee20857c7208ced7/userdata/shm major:0 minor:244 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a/volumes/kubernetes.io~projected/kube-api-access-ldwrf:{mountpoint:/var/lib/kubelet/pods/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a/volumes/kubernetes.io~projected/kube-api-access-ldwrf major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a/volumes/kubernetes.io~secret/cert major:0 minor:207 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/12134b5c-de72-4972-a185-3d5450c7b5e7/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/12134b5c-de72-4972-a185-3d5450c7b5e7/volumes/kubernetes.io~secret/agent-certs major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/volumes/kubernetes.io~projected/kube-api-access-cns2t:{mountpoint:/var/lib/kubelet/pods/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/volumes/kubernetes.io~projected/kube-api-access-cns2t major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:239 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/45b113ed-a156-4c53-bee5-26707e5717c3/volumes/kubernetes.io~projected/kube-api-access-5hqjj:{mountpoint:/var/lib/kubelet/pods/45b113ed-a156-4c53-bee5-26707e5717c3/volumes/kubernetes.io~projected/kube-api-access-5hqjj major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/569992a0-aa75-48cb-ad8d-fb61295a5093/volumes/kubernetes.io~projected/kube-api-access-4x66t:{mountpoint:/var/lib/kubelet/pods/569992a0-aa75-48cb-ad8d-fb61295a5093/volumes/kubernetes.io~projected/kube-api-access-4x66t major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/569992a0-aa75-48cb-ad8d-fb61295a5093/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/569992a0-aa75-48cb-ad8d-fb61295a5093/volumes/kubernetes.io~secret/metrics-certs major:0 minor:419 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5874f474-b442-4eec-a808-33997e342cb5/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/5874f474-b442-4eec-a808-33997e342cb5/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5874f474-b442-4eec-a808-33997e342cb5/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/5874f474-b442-4eec-a808-33997e342cb5/volumes/kubernetes.io~empty-dir/tmp major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5874f474-b442-4eec-a808-33997e342cb5/volumes/kubernetes.io~projected/kube-api-access-lw7vq:{mountpoint:/var/lib/kubelet/pods/5874f474-b442-4eec-a808-33997e342cb5/volumes/kubernetes.io~projected/kube-api-access-lw7vq major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/67d3a22c-930e-4113-ba61-2aecccabf9d2/volumes/kubernetes.io~projected/kube-api-access-8kc9p:{mountpoint:/var/lib/kubelet/pods/67d3a22c-930e-4113-ba61-2aecccabf9d2/volumes/kubernetes.io~projected/kube-api-access-8kc9p major:0 minor:425 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/72976414-f225-493b-9c55-3ab89bb831a8/volumes/kubernetes.io~projected/kube-api-access-bwsxf:{mountpoint:/var/lib/kubelet/pods/72976414-f225-493b-9c55-3ab89bb831a8/volumes/kubernetes.io~projected/kube-api-access-bwsxf major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b24f224-6dab-4478-890b-9da0d0679187/volumes/kubernetes.io~projected/kube-api-access-flf2g:{mountpoint:/var/lib/kubelet/pods/9b24f224-6dab-4478-890b-9da0d0679187/volumes/kubernetes.io~projected/kube-api-access-flf2g major:0 minor:343 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9b24f224-6dab-4478-890b-9da0d0679187/volumes/kubernetes.io~secret/monitoring-plugin-cert:{mountpoint:/var/lib/kubelet/pods/9b24f224-6dab-4478-890b-9da0d0679187/volumes/kubernetes.io~secret/monitoring-plugin-cert major:0 minor:342 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~projected/kube-api-access-rqrzv:{mountpoint:/var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~projected/kube-api-access-rqrzv major:0 minor:339 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:331 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:338 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:335 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2b8eb86-b9d8-4e70-8c4e-736070d94081/volumes/kubernetes.io~projected/kube-api-access-hwlmr:{mountpoint:/var/lib/kubelet/pods/a2b8eb86-b9d8-4e70-8c4e-736070d94081/volumes/kubernetes.io~projected/kube-api-access-hwlmr major:0 minor:242 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2b8eb86-b9d8-4e70-8c4e-736070d94081/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/a2b8eb86-b9d8-4e70-8c4e-736070d94081/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:238 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a2b8eb86-b9d8-4e70-8c4e-736070d94081/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/a2b8eb86-b9d8-4e70-8c4e-736070d94081/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:240 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8/volumes/kubernetes.io~projected/kube-api-access-z6gbn:{mountpoint:/var/lib/kubelet/pods/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8/volumes/kubernetes.io~projected/kube-api-access-z6gbn major:0 minor:412 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:411 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:406 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/web-config/alertmanager/9 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/web-config/config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~empty-dir/config-out major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~projected/kube-api-access-9cbmh:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~projected/kube-api-access-9cbmh major:0 minor:292 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~projected/tls-assets major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:286 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/config-volume major:0 minor:285 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:291 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:287 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:288 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:290 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/web-config major:0 minor:289 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8014993-7131-499c-98b3-57b1b7429729/volumes/kubernetes.io~projected/kube-api-access-5kmlq:{mountpoint:/var/lib/kubelet/pods/b8014993-7131-499c-98b3-57b1b7429729/volumes/kubernetes.io~projected/kube-api-access-5kmlq major:0 minor:399 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8014993-7131-499c-98b3-57b1b7429729/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/b8014993-7131-499c-98b3-57b1b7429729/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b8014993-7131-499c-98b3-57b1b7429729/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/b8014993-7131-499c-98b3-57b1b7429729/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be63e086-3851-459b-9123-262570be3b86/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/be63e086-3851-459b-9123-262570be3b86/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be63e086-3851-459b-9123-262570be3b86/volumes/kubernetes.io~projected/kube-api-access-ds4gb:{mountpoint:/var/lib/kubelet/pods/be63e086-3851-459b-9123-262570be3b86/volumes/kubernetes.io~projected/kube-api-access-ds4gb major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/be63e086-3851-459b-9123-262570be3b86/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/be63e086-3851-459b-9123-262570be3b86/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8e15255-91bb-411a-9c47-3c714735e70d/volumes/kubernetes.io~projected/kube-api-access-v87r5:{mountpoint:/var/lib/kubelet/pods/c8e15255-91bb-411a-9c47-3c714735e70d/volumes/kubernetes.io~projected/kube-api-access-v87r5 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c8e15255-91bb-411a-9c47-3c714735e70d/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/c8e15255-91bb-411a-9c47-3c714735e70d/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cb25da1d-2305-4f75-9920-e52fe596f5b7/volumes/kubernetes.io~projected/kube-api-access-cvtgq:{mountpoint:/var/lib/kubelet/pods/cb25da1d-2305-4f75-9920-e52fe596f5b7/volumes/kubernetes.io~projected/kube-api-access-cvtgq major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cc9755b7-6248-49a7-93c2-06a74060d379/volumes/kubernetes.io~projected/kube-api-access-qkbpx:{mountpoint:/var/lib/kubelet/pods/cc9755b7-6248-49a7-93c2-06a74060d379/volumes/kubernetes.io~projected/kube-api-access-qkbpx major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/cf4747b6-b5df-4e8e-a4f7-6de80acaad72/volumes/kubernetes.io~projected/kube-api-access-vn2zm:{mountpoint:/var/lib/kubelet/pods/cf4747b6-b5df-4e8e-a4f7-6de80acaad72/volumes/kubernetes.io~projected/kube-api-access-vn2zm major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1dc93b8-6bd0-484e-a06c-d5a659e44eba/volumes/kubernetes.io~projected/kube-api-access-2gnnr:{mountpoint:/var/lib/kubelet/pods/d1dc93b8-6bd0-484e-a06c-d5a659e44eba/volumes/kubernetes.io~projected/kube-api-access-2gnnr major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d1dc93b8-6bd0-484e-a06c-d5a659e44eba/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/d1dc93b8-6bd0-484e-a06c-d5a659e44eba/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da3c94aa-c880-4844-8cc8-2182ff86e321/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/da3c94aa-c880-4844-8cc8-2182ff86e321/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:424 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e01c7dab-d74b-4efe-9a0b-58a9eba6d956/volumes/kubernetes.io~projected/kube-api-access-qmztd:{mountpoint:/var/lib/kubelet/pods/e01c7dab-d74b-4efe-9a0b-58a9eba6d956/volumes/kubernetes.io~projected/kube-api-access-qmztd major:0 minor:237 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e01c7dab-d74b-4efe-9a0b-58a9eba6d956/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/e01c7dab-d74b-4efe-9a0b-58a9eba6d956/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:236 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e01c7dab-d74b-4efe-9a0b-58a9eba6d956/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/e01c7dab-d74b-4efe-9a0b-58a9eba6d956/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:232 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e22030b7-c861-47ee-b7d7-75f6d659e9c2/volumes/kubernetes.io~projected/kube-api-access-j2dv7:{mountpoint:/var/lib/kubelet/pods/e22030b7-c861-47ee-b7d7-75f6d659e9c2/volumes/kubernetes.io~projected/kube-api-access-j2dv7 major:0 minor:225 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e22030b7-c861-47ee-b7d7-75f6d659e9c2/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/e22030b7-c861-47ee-b7d7-75f6d659e9c2/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e22030b7-c861-47ee-b7d7-75f6d659e9c2/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/e22030b7-c861-47ee-b7d7-75f6d659e9c2/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:217 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e50ac2ed-9597-47ec-a4be-078d412f0b61/volumes/kubernetes.io~projected/kube-api-access-sgv82:{mountpoint:/var/lib/kubelet/pods/e50ac2ed-9597-47ec-a4be-078d412f0b61/volumes/kubernetes.io~projected/kube-api-access-sgv82 major:0 minor:66 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/1f18aa59a4c40b64bf10a44243b7d890cd09b18b4ad8ada87ba29d2030d88a91/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/4273ea392f32211ce9ecbe5d3ed9d4e0e1fad6f661f37b4df97c3aebac68bc86/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/b9e938bdf336ca0220f3ad4132e55e753ddd2d97a9644d068075bcc4f712da3d/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/11fe0fb9cc4aaa548df69b01e55fc360e4ed2333aa32f6e4e6fb4b5f08a9ab13/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/77bbc84402626e76cc603880801fb865a5847e0ba07666d9cb7507864c112e41/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/6333345823d8195ae2758012e32ee26b7165e7124b1475572f4b6d645374e309/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/b40089ba8356a99681d711460808ace2d489e4100e8c47fcee29bbef40c7d705/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/3cb08113d62215d0c043dd58c633a788b22485065b83e6fe13c4bee2401622a1/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/0660f615c24c962eb0d5afb02a6bd33f95e317e7d8506fe2a6ca189a4e465f3f/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/9d2b5e3a441414a94f2eaaa5714315c2866479648c77e694a860208a43eee5f5/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-147:{mountpoint:/var/lib/containers/storage/overlay/98e173e25ab7db9952b9e0ccd1a5d8ad355264e5ab68cd47474f4ed85b39cfb3/merged major:0 minor:147 fsType:overlay blockSize:0} overlay_0-148:{mountpoint:/var/lib/containers/storage/overlay/d4b7ea488931ea5aeae0bfc343562d84cbf04833aa1ea68966711457dac72918/merged major:0 minor:148 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/64056f672222664853162d53fccfbcc9a3cf2c497515b963f516d05bb75391df/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/088c20bfa5b52c8600a0d7486c4ea621163c0f32b81aa83135125085c089b571/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/93cf7e553de7313aced31b0d8889c230d15bbe641fd3f90d61c54e6bf0f56d44/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/bed257a6b1b24e6a904aa8d2a3cce8eeb85c82cfd195afef6c7faa18f97c7c51/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/53d1cd3b8a7f9388610110be78bd53bbfbe15ab6ef36e700ad81d5b288dad571/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6f5e2d2569741d5e27225f0821aa3be7cd27494c9857eca0793ec90e0ce8036c/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1756062f2f8464f632462478326651ca23f0ad4125cbfbd554ab361094386c71/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c170f04c97548dd77cd0ac93dc642148b17e648fe8394cc8d96d5d163ed4ad95/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/228660fcadbfb0ceb69f6dad9cf78f6b8c39b78478b1e99c26568796ca0fbc71/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/507cabad3218338a963146c1732d89fce3efaf4c2ab6ebaedf019c997fb2981c/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/c6966a2a98a63c1f9c3791c65c917909879ce775cb76dc17e8b4c2be553f41ec/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/0494ff3af63ec4a61e37ceaa7a4588729cced07debc65c13186fb71f51e96a97/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-228:{mountpoint:/var/lib/containers/storage/overlay/5924a422ff18e71595ff331369332ddb858af9348de6a2a957789bbebf871de5/merged major:0 minor:228 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/9aeb521b30e710bc680abaa78da6dd3684015f16cb2145e8a2cef019dd85e4bb/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/3357222a3b11754894c678dd8c7593df23128d8498f2975740e2fda64e7bf57b/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/f8fcfddcfdf35f8669b3b277c69a11d7e733eb7cbf1d2d0a37b876ccf8404c16/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/bd5425ec133d53e1a4cdaa8d8f13ab5ba0c3d550d4fabccfd3cbd1b3c8619636/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-256:{mountpoint:/var/lib/containers/storage/overlay/f4c039889e4defdd1b6bc1f5aad71998ec5efd8ba3183a58cf15ec3a9ce88c47/merged major:0 minor:256 fsType:overlay blockSize:0} overlay_0-257:{mountpoint:/var/lib/containers/storage/overlay/d6998bbba546a1cfa2ec66d406b4a705aa1bed9161fe2e5bbda5a1e7349416c9/merged major:0 minor:257 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/ab55cc3d0425eda8324a350925bc4509ea2e2a1af65db0c0e6c973983eddf31f/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-268:{mountpoint:/var/lib/containers/storage/overlay/19e26a12c1bb4b2c8fb594757d5a5121f063fe24d82d967083fb75eeac289aa9/merged major:0 minor:268 fsType:overlay blockSize:0} overlay_0-275:{mountpoint:/var/lib/containers/storage/overlay/12a15118c1f08d806bce03a220121986477e2fb928855acb4c8ddc65a660949c/merged major:0 minor:275 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/a80cc11a20222ffd13e157432abd354b4f64b65aa232c961abfce2d13159e6b0/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/4cde2b4052248ca52abc206606143a126001c0a61b4a5b981940a6ff80ec9e92/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-299:{mountpoint:/var/lib/containers/storage/overlay/f12862621616d107f0dd7947a09875f0b054fa8e05b34d982f5b955d8c0db498/merged major:0 minor:299 fsType:overlay blockSize:0} overlay_0-304:{mountpoint:/var/lib/containers/storage/overlay/0112242167b6ee33ddf8a6521467bb38bc5b542b52968686361b580fc2ac5dfc/merged major:0 minor:304 fsType:overlay blockSize:0} overlay_0-306:{mountpoint:/var/lib/containers/storage/overlay/331d29a0dfbf2f952f5006dde0427035230d25064f3e0e20aae00f4e2eb18a1f/merged major:0 minor:306 fsType:overlay blockSize:0} overlay_0-316:{mountpoint:/var/lib/containers/storage/overlay/d82c8bf523e52da6614b5395dc677435488296dd8c4b1ccc4b861598b1a0fe69/merged major:0 minor:316 fsType:overlay blockSize:0} overlay_0-321:{mountpoint:/var/lib/containers/storage/overlay/893326086e9070df47a7c7ce9e357aa2d2803d3dfc654a9dd6ce081bd032cba9/merged major:0 minor:321 fsType:overlay blockSize:0} overlay_0-326:{mountpoint:/var/lib/containers/storage/overlay/cd551ef0a40332d73308572aafae90c0ac78bc870c757efd8ce80fa26c6d3279/merged major:0 minor:326 fsType:overlay blockSize:0} overlay_0-336:{mountpoint:/var/lib/containers/storage/overlay/52cdc2308b58514fb5965f58caa0d8c468d329c69273e1323b34c1b0eebb3cab/merged major:0 minor:336 fsType:overlay blockSize:0} overlay_0-348:{mountpoint:/var/lib/containers/storage/overlay/208dbdf0ad6bc7fbb1472276ddc3cbb667a6ee8dbdb42147443d45ca45e597fe/merged major:0 minor:348 fsType:overlay blockSize:0} overlay_0-352:{mountpoint:/var/lib/containers/storage/overlay/d2845d381ad6124266377dd982d21f5a6ac4de70d1e9b9e1d93d40e627ba8950/merged major:0 minor:352 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/73e97b9ada1417638ed6beb7dd5fc89775356b90f629229b7b0970aa7aeac51b/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-356:{mountpoint:/var/lib/containers/storage/overlay/f9f25793335c77217017c2d6c3bd91a26cfcd8be3948501520e55ec05643c3b9/merged major:0 minor:356 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/9504e0be0f1a69ab4f52ff9917914840dd2131be38e288437d7b5d00064a74a5/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-367:{mountpoint:/var/lib/containers/storage/overlay/883c86df34b10dc42ecc9702e77e4e9afd41aba13e464aadba7e45a52458aec3/merged major:0 minor:367 fsType:overlay blockSize:0} overlay_0-372:{mountpoint:/var/lib/containers/storage/overlay/ce27407cb002fbe718bc9b50fed6b8ba78ab1016a9dedb0cdbaf1c2b45addfe0/merged major:0 minor:372 fsType:overlay blockSize:0} overlay_0-377:{mountpoint:/var/lib/containers/storage/overlay/844dfdd20abf7d58ca8aa3b5054111f19faa33d4e8ded0fc56afb81b0c9dcc9d/merged major:0 minor:377 fsType:overlay blockSize:0} overlay_0-382:{mountpoint:/var/lib/containers/storage/overlay/a36a309e4ec094ee4e90a12db31d837d0f8df6627068f8f89908ae29d3a894d8/merged major:0 minor:382 fsType:overlay blockSize:0} overlay_0-384:{mountpoint:/var/lib/containers/storage/overlay/9dab38c7a15656234e9c5fa37770f912462b5eb8204ffaf37f408c2b47c65bab/merged major:0 minor:384 fsType:overlay blockSize:0} overlay_0-402:{mountpoint:/var/lib/containers/storage/overlay/964d17b27aa313e3d9ca8fd029f45aee901828f6af6f3bd80a6270d6d32f49b4/merged major:0 minor:402 fsType:overlay blockSize:0} overlay_0-404:{mountpoint:/var/lib/containers/storage/overlay/4395afdda84fa9193a7dc60796e221e952048d6b980d3aea17e844cd5e972b13/merged major:0 minor:404 fsType:overlay blockSize:0} overlay_0-415:{mountpoint:/var/lib/containers/storage/overlay/eec65edbc195da396ee4082b90d82dc48aba31391091aa23f894fdf4e1740f94/merged major:0 minor:415 fsType:overlay blockSize:0} overlay_0-417:{mountpoint:/var/lib/containers/storage/overlay/2557b1c9af2fe34b58258040812d6ed64b8ab3f3fddfbdc7dd262ecc835b63c7/merged major:0 minor:417 fsType:overlay blockSize:0} overlay_0-432:{mountpoint:/var/lib/containers/storage/overlay/6a80c6a7197733775ae096a74b5c7cc6efb15e2e418a3cdda58bfedccc52bf6a/merged major:0 minor:432 fsType:overlay blockSize:0} overlay_0-434:{mountpoint:/var/lib/containers/storage/overlay/5dab93a6ca64d437a6071f3edc9a5b58e2ea55f5ad5d1e2e20efad601200f170/merged major:0 minor:434 fsType:overlay blockSize:0} overlay_0-436:{mountpoint:/var/lib/containers/storage/overlay/15694ac4ac9f920ef33e72bb1001c4ee04da22998f541e518613fd4f2ff5f6af/merged major:0 minor:436 fsType:overlay blockSize:0} overlay_0-438:{mountpoint:/var/lib/containers/storage/overlay/5e0c055865f816521e5dd7e50e0f6b5bff951bbc5114c27ca280abc8faee555e/merged major:0 minor:438 fsType:overlay blockSize:0} overlay_0-440:{mountpoint:/var/lib/containers/storage/overlay/1e06dee33e31f962177dc784dbb1e20eb7371c9d4042d9d2680b0d5fdbf3c7f8/merged major:0 minor:440 fsType:overlay blockSize:0} overlay_0-446:{mountpoint:/var/lib/containers/storage/overlay/d118f2294f592a0820aaa74a86bea29a2a7710e762f1c550aca30cad05abbc1e/merged major:0 minor:446 fsType:overlay blockSize:0} overlay_0-451:{mountpoint:/var/lib/containers/storage/overlay/c329682c9cc1dd610cf1d6e1920fcc12446ba53e323eac6a7e9ff7faa46de2be/merged major:0 minor:451 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/6a6b080835237082b73e7f9d325f80bb6f981d0702cb72c9dac3a3c4c8fee56a/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/e6825fa8ee467cfbe3d99a6e6b315905751922be3d6a090695480e9ba027b426/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-68:{mountpoint:/var/lib/containers/storage/overlay/fac92cb0e1847c585597b1cc4a1d2def21f1d49ab342609bb495a406aad8cf38/merged major:0 minor:68 fsType:overlay blockSize:0} overlay_0-72:{mountpoint:/var/lib/containers/storage/overlay/6770c3eb7298edea660b753ecdfebc6ea08b06bfef6deb33f407afcab0a80d4d/merged major:0 minor:72 fsType:overlay blockSize:0} overlay_0-78:{mountpoint:/var/lib/containers/storage/overlay/da9d3242402ef4fa44425715191fca44f771d27255f07d3cec58906357963ce5/merged major:0 minor:78 fsType:overlay blockSize:0} overlay_0-81:{mountpoint:/var/lib/containers/storage/overlay/5de875c82277fb574f6b0d37ec3c6814d8b34f47f0dcd7bbf98fba137bca2f81/merged major:0 minor:81 fsType:overlay blockSize:0} overlay_0-85:{mountpoint:/var/lib/containers/storage/overlay/432034413befcda344a9361968ef6bf95ac7b370ef0e98c70d6ad7f25438a5fe/merged major:0 minor:85 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/a5819f59b3c50a84262be4605662e04af413d021a31d9fa4329df63ca4079fb8/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/8bd4d2563fa28580db3cffaf9167d1871d455ce77136cdb364685028c5635dc3/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/1787155bee1651cfb800907efd12542ecb7127693bd2f2c00952c85af37d5e31/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/4d1a5b9f3d61979a3d11f028a1679c2e0b3ef85fcd09a21c9c88593000652535/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 16:21:30.909093 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.908666 6400 manager.go:217] Machine: {Timestamp:2026-10-30 16:21:30.908192793 +0000 UTC m=+0.058473772 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3068967 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e7bb25d1b648c472416a203ec6f1a SystemUUID:ec2e7bb2-5d1b-648c-4724-16a203ec6f1a BootID:32273130-ad91-492e-8ece-9ba5f5249a7d Filesystems:[{Device:/run/containers/storage/overlay-containers/07d57a9af5deb83e3e36e68a621dec198a98234087076fcea3f17542366a4eb5/userdata/shm DeviceMajor:0 DeviceMinor:203 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/12134b5c-de72-4972-a185-3d5450c7b5e7/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5bc7b0b7b31bf53d7846f8be2ee9ffd5caa807beed7aa0c61800119b74253beb/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-257 DeviceMajor:0 DeviceMinor:257 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-299 DeviceMajor:0 DeviceMinor:299 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-68 DeviceMajor:0 DeviceMinor:68 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e01c7dab-d74b-4efe-9a0b-58a9eba6d956/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:232 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-326 DeviceMajor:0 DeviceMinor:326 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a74ad2aa4355d3eeb56a53a285b0325202d10f7296bbf4436e85a22a4a76007e/userdata/shm DeviceMajor:0 DeviceMinor:400 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-415 DeviceMajor:0 DeviceMinor:415 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-417 DeviceMajor:0 DeviceMinor:417 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:287 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/371dd94fe0177fcf78a797a514dd4cbfba3ccc8d00c215f2e90734fbd5df3ad9/userdata/shm DeviceMajor:0 DeviceMinor:293 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-321 DeviceMajor:0 DeviceMinor:321 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-436 DeviceMajor:0 DeviceMinor:436 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:291 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-352 DeviceMajor:0 DeviceMinor:352 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8014993-7131-499c-98b3-57b1b7429729/volumes/kubernetes.io~projected/kube-api-access-5kmlq DeviceMajor:0 DeviceMinor:399 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-438 DeviceMajor:0 DeviceMinor:438 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-446 DeviceMajor:0 DeviceMinor:446 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/45b113ed-a156-4c53-bee5-26707e5717c3/volumes/kubernetes.io~projected/kube-api-access-5hqjj DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/be63e086-3851-459b-9123-262570be3b86/volumes/kubernetes.io~projected/kube-api-access-ds4gb DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e22030b7-c861-47ee-b7d7-75f6d659e9c2/volumes/kubernetes.io~projected/kube-api-access-j2dv7 DeviceMajor:0 DeviceMinor:225 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-372 DeviceMajor:0 DeviceMinor:372 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-434 DeviceMajor:0 DeviceMinor:434 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/55e377cd2543dde7f785cffa2dbd8e0a59f376979a1b06d430400698f1e51f5b/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:288 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-356 DeviceMajor:0 DeviceMinor:356 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-384 DeviceMajor:0 DeviceMinor:384 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:207 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-275 DeviceMajor:0 DeviceMinor:275 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2b8eb86-b9d8-4e70-8c4e-736070d94081/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:240 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:331 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/be63e086-3851-459b-9123-262570be3b86/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:280 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/67d3a22c-930e-4113-ba61-2aecccabf9d2/volumes/kubernetes.io~projected/kube-api-access-8kc9p DeviceMajor:0 DeviceMinor:425 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-451 DeviceMajor:0 DeviceMinor:451 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e50ac2ed-9597-47ec-a4be-078d412f0b61/volumes/kubernetes.io~projected/kube-api-access-sgv82 DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/e01c7dab-d74b-4efe-9a0b-58a9eba6d956/volumes/kubernetes.io~projected/kube-api-access-qmztd DeviceMajor:0 DeviceMinor:237 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a42f88609e94dc9ee337ffff0f37772fbaef9f59f4cdf4df369a135c2a1bba4c/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8c8175c5ed0e3f758f1224bb024bd1c027e3efdc79d3a8f2b8d4516b86fcc475/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/d1dc93b8-6bd0-484e-a06c-d5a659e44eba/volumes/kubernetes.io~projected/kube-api-access-2gnnr DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5203e1370fb8a24e66347c1bd6dc51eed70d0103db315fa69856ca17b6ea0502/userdata/shm DeviceMajor:0 DeviceMinor:246 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-304 DeviceMajor:0 DeviceMinor:304 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dad6a8a9012b28ad43afa0d0739af1dc300a4ad8ba17f721a039d42d8e0a3e28/userdata/shm DeviceMajor:0 DeviceMinor:426 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/var/lib/kubelet/pods/e01c7dab-d74b-4efe-9a0b-58a9eba6d956/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:236 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/volumes/kubernetes.io~projected/kube-api-access-cns2t DeviceMajor:0 DeviceMinor:243 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-440 DeviceMajor:0 DeviceMinor:440 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2b8eb86-b9d8-4e70-8c4e-736070d94081/volumes/kubernetes.io~projected/kube-api-access-hwlmr DeviceMajor:0 DeviceMinor:242 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-268 DeviceMajor:0 DeviceMinor:268 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-316 DeviceMajor:0 DeviceMinor:316 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/569992a0-aa75-48cb-ad8d-fb61295a5093/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:419 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ca9666220a2ed5ac47c7b925b0a95bb4cf61ddd0852cd2c5aa9a93f01163b338/userdata/shm DeviceMajor:0 DeviceMinor:340 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/09c1ac4d7705d3057cc77a8993d87c622e4d54efb2e60124b7819a6fa0f53fe3/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d1dc93b8-6bd0-484e-a06c-d5a659e44eba/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/06f0c2cfa5ed9fff43f86ce77efaaab23ac240158b23d4a3f41d9713ea796c3c/userdata/shm DeviceMajor:0 DeviceMinor:226 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-256 DeviceMajor:0 DeviceMinor:256 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-432 DeviceMajor:0 DeviceMinor:432 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8e15255-91bb-411a-9c47-3c714735e70d/volumes/kubernetes.io~projected/kube-api-access-v87r5 DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-336 DeviceMajor:0 DeviceMinor:336 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e068c96992dad3f8ef6035a37c77dd7ddf57157a437c388c4a5573b1bd53a9a9/userdata/shm DeviceMajor:0 DeviceMinor:74 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8dd102e070f9a4d8557721bedc77be7ee8a47857446863ee6f9aea82af9ecdc/userdata/shm DeviceMajor:0 DeviceMinor:83 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffc76cbf645fbace71ad36296e14f50e9dbbc00f6cc73378ee20857c7208ced7/userdata/shm DeviceMajor:0 DeviceMinor:244 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-306 DeviceMajor:0 DeviceMinor:306 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:338 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5b9c936fbf7990544e33c048a35d79575c507dbc22ae6732a20e8c5bff813be1/userdata/shm DeviceMajor:0 DeviceMinor:428 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c480f118ec53bbba0e7858f55baeb2a3f1cf76b4ded3fd85b6d150ff40522aaa/userdata/shm DeviceMajor:0 DeviceMinor:70 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/32da6cefad5f9764cf3178c7c6878e4ded6d15d81574464735d04eb0da6a5703/userdata/shm DeviceMajor:0 DeviceMinor:413 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dc70e5caa9a93c9ec992515edfba0352a26ee50ac8667eecbcfabb922185ac8e/userdata/shm DeviceMajor:0 DeviceMinor:430 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/569992a0-aa75-48cb-ad8d-fb61295a5093/volumes/kubernetes.io~projected/kube-api-access-4x66t DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f696498181b3360832b434fed9f73d14e4a751195dcd8d9c3c64e7dd57ce6ffc/userdata/shm DeviceMajor:0 DeviceMinor:100 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:286 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-404 DeviceMajor:0 DeviceMinor:404 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-81 DeviceMajor:0 DeviceMinor:81 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:285 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:335 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-402 DeviceMajor:0 DeviceMinor:402 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:411 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/dd3682e2ac5a533d0de34eb03be249a7edb22151f8465e37daa92fcc63ce6a76/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-382 DeviceMajor:0 DeviceMinor:382 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-78 DeviceMajor:0 DeviceMinor:78 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-148 DeviceMajor:0 DeviceMinor:148 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:406 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/da3c94aa-c880-4844-8cc8-2182ff86e321/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:424 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6d0ffdfc11a08f5cc3957740a7e3ef60833c74db082289fa4e381f10835cc270/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/cb25da1d-2305-4f75-9920-e52fe596f5b7/volumes/kubernetes.io~projected/kube-api-access-cvtgq DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a/volumes/kubernetes.io~projected/kube-api-access-ldwrf DeviceMajor:0 DeviceMinor:208 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b8014993-7131-499c-98b3-57b1b7429729/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:398 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5874f474-b442-4eec-a808-33997e342cb5/volumes/kubernetes.io~projected/kube-api-access-lw7vq DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a869f354c2abbc63e2aba5d9a42b0388132aa0f561bbfd6477624e707347ae4b/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0af2178deb53c5d43e8b86ff93686a2b479c30923bb0969a9ca457653639e35/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-85 DeviceMajor:0 DeviceMinor:85 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a2b8eb86-b9d8-4e70-8c4e-736070d94081/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:238 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b24f224-6dab-4478-890b-9da0d0679187/volumes/kubernetes.io~projected/kube-api-access-flf2g DeviceMajor:0 DeviceMinor:343 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-348 DeviceMajor:0 DeviceMinor:348 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5874f474-b442-4eec-a808-33997e342cb5/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-147 DeviceMajor:0 DeviceMinor:147 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b8014993-7131-499c-98b3-57b1b7429729/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:393 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c8e15255-91bb-411a-9c47-3c714735e70d/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:289 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6cdafef6533e60bb7f0c68a6000af6a1432653a56012191796e66690a7ee449d/userdata/shm DeviceMajor:0 DeviceMinor:350 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-367 DeviceMajor:0 DeviceMinor:367 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e22030b7-c861-47ee-b7d7-75f6d659e9c2/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:217 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~projected/kube-api-access-9cbmh DeviceMajor:0 DeviceMinor:292 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9f3cf821-229a-4690-abad-332006a3882a/volumes/kubernetes.io~projected/kube-api-access-rqrzv DeviceMajor:0 DeviceMinor:339 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/72976414-f225-493b-9c55-3ab89bb831a8/volumes/kubernetes.io~projected/kube-api-access-bwsxf DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-228 DeviceMajor:0 DeviceMinor:228 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:290 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-377 DeviceMajor:0 DeviceMinor:377 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8/volumes/kubernetes.io~projected/kube-api-access-z6gbn DeviceMajor:0 DeviceMinor:412 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cf4747b6-b5df-4e8e-a4f7-6de80acaad72/volumes/kubernetes.io~projected/kube-api-access-vn2zm DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/cc9755b7-6248-49a7-93c2-06a74060d379/volumes/kubernetes.io~projected/kube-api-access-qkbpx DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e22030b7-c861-47ee-b7d7-75f6d659e9c2/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:224 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/be63e086-3851-459b-9123-262570be3b86/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:239 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/9b24f224-6dab-4478-890b-9da0d0679187/volumes/kubernetes.io~secret/monitoring-plugin-cert DeviceMajor:0 DeviceMinor:342 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5874f474-b442-4eec-a808-33997e342cb5/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-72 DeviceMajor:0 DeviceMinor:72 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:06f0c2cfa5ed9ff MacAddress:fe:ef:a8:32:b8:96 Speed:10000 Mtu:8901} {Name:07d57a9af5deb83 MacAddress:5e:fb:7b:bc:76:16 Speed:10000 Mtu:8901} {Name:32da6cefad5f976 MacAddress:96:61:58:cb:c7:2c Speed:10000 Mtu:8901} {Name:371dd94fe0177fc MacAddress:ce:1d:55:56:08:f3 Speed:10000 Mtu:8901} {Name:55e377cd2543dde MacAddress:9e:2d:7a:64:a1:ac Speed:10000 Mtu:8901} {Name:5b9c936fbf79905 MacAddress:ae:f8:d8:38:c6:bb Speed:10000 Mtu:8901} {Name:5bc7b0b7b31bf53 MacAddress:7a:71:77:69:fa:9e Speed:10000 Mtu:8901} {Name:6cdafef6533e60b MacAddress:16:12:f2:24:97:76 Speed:10000 Mtu:8901} {Name:a74ad2aa4355d3e MacAddress:26:88:f6:e7:52:4e Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:dd:52:16:bd Speed:0 Mtu:9001} {Name:br-int MacAddress:46:98:5a:a6:4d:3f Speed:0 Mtu:8901} {Name:ca9666220a2ed5a MacAddress:4a:f7:ff:6a:7d:37 Speed:10000 Mtu:8901} {Name:dad6a8a9012b28a MacAddress:fe:07:e5:58:07:64 Speed:10000 Mtu:8901} {Name:dc70e5caa9a93c9 MacAddress:ae:ca:05:9d:e9:9f Speed:10000 Mtu:8901} {Name:dd3682e2ac5a533 MacAddress:e6:70:49:b4:66:ca Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:dd:52:16:bd Speed:0 Mtu:9001} {Name:ffc76cbf645fbac MacAddress:52:50:61:f0:9d:70 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:fe:05:c5:dd:0a:e5 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:6a:2e:a1:99:ec:78 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 16:21:30.909308 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.909097 6400 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 16:21:30.909308 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.909143 6400 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 16:21:30.909501 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.909473 6400 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 16:21:30.909643 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.909503 6400 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-128-181.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 16:21:30.909685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.909645 6400 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 16:21:30.909685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.909653 6400 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 16:21:30.909685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.909664 6400 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:21:30.909685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.909678 6400 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 16:21:30.909789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.909770 6400 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 16:21:30.909931 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.909921 6400 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 16:21:30.910038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.910029 6400 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 16:21:30.910073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.910065 6400 kubelet.go:493] "Attempting to sync node with API server" Oct 30 16:21:30.910101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.910076 6400 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 16:21:30.910101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.910088 6400 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 16:21:30.910101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.910097 6400 kubelet.go:405] "Adding apiserver pod source" Oct 30 16:21:30.910177 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.910105 6400 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 16:21:30.910973 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.910958 6400 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:21:30.911022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.910978 6400 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:21:30.911517 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.911498 6400 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 16:21:30.914511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914496 6400 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 16:21:30.914511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914511 6400 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 16:21:30.914634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914618 6400 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 16:21:30.914804 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914793 6400 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 16:21:30.914834 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914822 6400 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 16:21:30.914914 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914906 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 16:21:30.914944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914921 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 16:21:30.914944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914927 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 16:21:30.914944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914932 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 16:21:30.914944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914939 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 16:21:30.914944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914944 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 16:21:30.915084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914950 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 16:21:30.915084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914956 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 16:21:30.915084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914963 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 16:21:30.915084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914969 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 16:21:30.915084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.914997 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 16:21:30.915084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.915007 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 16:21:30.915084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.915033 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 16:21:30.915084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.915038 6400 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 16:21:30.918406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.918393 6400 server.go:1263] "Started kubelet" Oct 30 16:21:30.918835 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.918470 6400 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 16:21:30.918835 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.918507 6400 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 16:21:30.918835 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.918625 6400 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 16:21:30.919418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.918934 6400 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 16:21:30.919558 ip-10-0-128-181 systemd[1]: Started Kubernetes Kubelet. Oct 30 16:21:30.923324 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.922993 6400 server.go:317] "Adding debug handlers to kubelet server" Oct 30 16:21:30.926548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.925552 6400 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 16:21:30.926548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.925571 6400 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 16:21:30.926548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.925684 6400 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 16:21:30.926548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.925695 6400 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 16:21:30.926548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.925673 6400 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 16:15:21 +0000 UTC" deadline="2028-07-30 15:01:07.656171294 +0000 UTC" Oct 30 16:21:30.926548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.925743 6400 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15334h39m36.73043257s" Oct 30 16:21:30.926548 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:30.925751 6400 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-128-181.ec2.internal\" not found" Oct 30 16:21:30.926548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.925835 6400 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 16:21:30.928141 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928121 6400 factory.go:55] Registering systemd factory Oct 30 16:21:30.928141 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928141 6400 factory.go:223] Registration of the systemd container factory successfully Oct 30 16:21:30.928402 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928370 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e01c7dab-d74b-4efe-9a0b-58a9eba6d956" volumeName="kubernetes.io/projected/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-kube-api-access-qmztd" seLinuxMountContext="" Oct 30 16:21:30.928464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928415 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e22030b7-c861-47ee-b7d7-75f6d659e9c2" volumeName="kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-service-ca" seLinuxMountContext="" Oct 30 16:21:30.928464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928429 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f3cf821-229a-4690-abad-332006a3882a" volumeName="kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 16:21:30.928464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928441 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" volumeName="kubernetes.io/projected/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-kube-api-access-z6gbn" seLinuxMountContext="" Oct 30 16:21:30.928634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928463 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" volumeName="kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-serving-cert" seLinuxMountContext="" Oct 30 16:21:30.928634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928474 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 16:21:30.928634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928485 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8014993-7131-499c-98b3-57b1b7429729" volumeName="kubernetes.io/empty-dir/b8014993-7131-499c-98b3-57b1b7429729-tmp" seLinuxMountContext="" Oct 30 16:21:30.928634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928508 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45b113ed-a156-4c53-bee5-26707e5717c3" volumeName="kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-cni-binary-copy" seLinuxMountContext="" Oct 30 16:21:30.928634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928524 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2b8eb86-b9d8-4e70-8c4e-736070d94081" volumeName="kubernetes.io/projected/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-api-access-hwlmr" seLinuxMountContext="" Oct 30 16:21:30.928634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928537 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 16:21:30.928634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928572 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e15255-91bb-411a-9c47-3c714735e70d" volumeName="kubernetes.io/empty-dir/c8e15255-91bb-411a-9c47-3c714735e70d-tmp-dir" seLinuxMountContext="" Oct 30 16:21:30.928634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928583 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e22030b7-c861-47ee-b7d7-75f6d659e9c2" volumeName="kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-config" seLinuxMountContext="" Oct 30 16:21:30.928634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928593 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a" volumeName="kubernetes.io/projected/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-kube-api-access-cns2t" seLinuxMountContext="" Oct 30 16:21:30.928634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928637 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f3cf821-229a-4690-abad-332006a3882a" volumeName="kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-client-ca-bundle" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928659 6400 factory.go:153] Registering CRI-O factory Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928651 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2b8eb86-b9d8-4e70-8c4e-736070d94081" volumeName="kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-metrics-client-ca" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928677 6400 factory.go:223] Registration of the crio container factory successfully Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928681 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" volumeName="kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-oauth-serving-cert" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928703 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12134b5c-de72-4972-a185-3d5450c7b5e7" volumeName="kubernetes.io/configmap/12134b5c-de72-4972-a185-3d5450c7b5e7-konnectivity-ca" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928715 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="12134b5c-de72-4972-a185-3d5450c7b5e7" volumeName="kubernetes.io/secret/12134b5c-de72-4972-a185-3d5450c7b5e7-agent-certs" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928726 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45b113ed-a156-4c53-bee5-26707e5717c3" volumeName="kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-multus-daemon-config" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928734 6400 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928741 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f3cf821-229a-4690-abad-332006a3882a" volumeName="kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928755 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2b8eb86-b9d8-4e70-8c4e-736070d94081" volumeName="kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928767 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" volumeName="kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-trusted-ca-bundle" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928786 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" volumeName="kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-service-ca" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928800 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-web-config" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928812 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-main-db" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928826 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8014993-7131-499c-98b3-57b1b7429729" volumeName="kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-console-config" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928840 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" volumeName="kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-binary-copy" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928855 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1dc93b8-6bd0-484e-a06c-d5a659e44eba" volumeName="kubernetes.io/projected/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-api-access-2gnnr" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928877 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a" volumeName="kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928890 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="569992a0-aa75-48cb-ad8d-fb61295a5093" volumeName="kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928904 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928757 6400 factory.go:103] Registering Raw factory Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928957 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8014993-7131-499c-98b3-57b1b7429729" volumeName="kubernetes.io/projected/b8014993-7131-499c-98b3-57b1b7429729-kube-api-access-5kmlq" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928989 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be63e086-3851-459b-9123-262570be3b86" volumeName="kubernetes.io/projected/be63e086-3851-459b-9123-262570be3b86-kube-api-access-ds4gb" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.928993 6400 manager.go:1199] Started watching for new ooms in manager Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929025 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e01c7dab-d74b-4efe-9a0b-58a9eba6d956" volumeName="kubernetes.io/configmap/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-metrics-client-ca" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929047 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50ac2ed-9597-47ec-a4be-078d412f0b61" volumeName="kubernetes.io/projected/e50ac2ed-9597-47ec-a4be-078d412f0b61-kube-api-access-sgv82" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929065 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="569992a0-aa75-48cb-ad8d-fb61295a5093" volumeName="kubernetes.io/projected/569992a0-aa75-48cb-ad8d-fb61295a5093-kube-api-access-4x66t" seLinuxMountContext="" Oct 30 16:21:30.929058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929075 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72976414-f225-493b-9c55-3ab89bb831a8" volumeName="kubernetes.io/configmap/72976414-f225-493b-9c55-3ab89bb831a8-serviceca" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929086 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f3cf821-229a-4690-abad-332006a3882a" volumeName="kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929094 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" volumeName="kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-oauth-config" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929102 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-cluster-tls-config" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929111 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb25da1d-2305-4f75-9920-e52fe596f5b7" volumeName="kubernetes.io/empty-dir/cb25da1d-2305-4f75-9920-e52fe596f5b7-tmp-dir" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929121 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1dc93b8-6bd0-484e-a06c-d5a659e44eba" volumeName="kubernetes.io/configmap/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929131 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a" volumeName="kubernetes.io/secret/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-cert" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929139 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="67d3a22c-930e-4113-ba61-2aecccabf9d2" volumeName="kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929145 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="72976414-f225-493b-9c55-3ab89bb831a8" volumeName="kubernetes.io/projected/72976414-f225-493b-9c55-3ab89bb831a8-kube-api-access-bwsxf" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929152 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" volumeName="kubernetes.io/empty-dir/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-tmp" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929170 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-out" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929177 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8014993-7131-499c-98b3-57b1b7429729" volumeName="kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-service-ca" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929192 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8014993-7131-499c-98b3-57b1b7429729" volumeName="kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-serving-cert" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929201 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e01c7dab-d74b-4efe-9a0b-58a9eba6d956" volumeName="kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929208 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b24f224-6dab-4478-890b-9da0d0679187" volumeName="kubernetes.io/projected/9b24f224-6dab-4478-890b-9da0d0679187-kube-api-access-flf2g" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929216 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f3cf821-229a-4690-abad-332006a3882a" volumeName="kubernetes.io/projected/9f3cf821-229a-4690-abad-332006a3882a-kube-api-access-rqrzv" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929223 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-metrics-client-ca" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929230 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-kube-api-access-9cbmh" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929237 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be63e086-3851-459b-9123-262570be3b86" volumeName="kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-config" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929246 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be63e086-3851-459b-9123-262570be3b86" volumeName="kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-env-overrides" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929296 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da3c94aa-c880-4844-8cc8-2182ff86e321" volumeName="kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929306 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e22030b7-c861-47ee-b7d7-75f6d659e9c2" volumeName="kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-serving-cert" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929314 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9b24f224-6dab-4478-890b-9da0d0679187" volumeName="kubernetes.io/secret/9b24f224-6dab-4478-890b-9da0d0679187-monitoring-plugin-cert" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929321 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be63e086-3851-459b-9123-262570be3b86" volumeName="kubernetes.io/secret/be63e086-3851-459b-9123-262570be3b86-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929328 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1dc93b8-6bd0-484e-a06c-d5a659e44eba" volumeName="kubernetes.io/secret/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929337 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e01c7dab-d74b-4efe-9a0b-58a9eba6d956" volumeName="kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929344 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e22030b7-c861-47ee-b7d7-75f6d659e9c2" volumeName="kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-oauth-serving-cert" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929352 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e22030b7-c861-47ee-b7d7-75f6d659e9c2" volumeName="kubernetes.io/empty-dir/e22030b7-c861-47ee-b7d7-75f6d659e9c2-tmp" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929360 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2b8eb86-b9d8-4e70-8c4e-736070d94081" volumeName="kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929377 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8014993-7131-499c-98b3-57b1b7429729" volumeName="kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-oauth-serving-cert" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929384 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e15255-91bb-411a-9c47-3c714735e70d" volumeName="kubernetes.io/configmap/c8e15255-91bb-411a-9c47-3c714735e70d-config-volume" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929386 6400 manager.go:319] Starting recovery of all containers Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929392 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" volumeName="kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929402 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d1dc93b8-6bd0-484e-a06c-d5a659e44eba" volumeName="kubernetes.io/empty-dir/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-data-volume" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929417 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e50ac2ed-9597-47ec-a4be-078d412f0b61" volumeName="kubernetes.io/configmap/e50ac2ed-9597-47ec-a4be-078d412f0b61-iptables-alerter-script" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929425 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5874f474-b442-4eec-a808-33997e342cb5" volumeName="kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-etc-tuned" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929432 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f3cf821-229a-4690-abad-332006a3882a" volumeName="kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929440 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2b8eb86-b9d8-4e70-8c4e-736070d94081" volumeName="kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929448 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" volumeName="kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-config" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929455 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-volume" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929462 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8014993-7131-499c-98b3-57b1b7429729" volumeName="kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-trusted-ca-bundle" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929469 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e22030b7-c861-47ee-b7d7-75f6d659e9c2" volumeName="kubernetes.io/projected/e22030b7-c861-47ee-b7d7-75f6d659e9c2-kube-api-access-j2dv7" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929477 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a" volumeName="kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929484 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="45b113ed-a156-4c53-bee5-26707e5717c3" volumeName="kubernetes.io/projected/45b113ed-a156-4c53-bee5-26707e5717c3-kube-api-access-5hqjj" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929492 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a" volumeName="kubernetes.io/projected/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-kube-api-access-ldwrf" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929499 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5874f474-b442-4eec-a808-33997e342cb5" volumeName="kubernetes.io/projected/5874f474-b442-4eec-a808-33997e342cb5-kube-api-access-lw7vq" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929506 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="9f3cf821-229a-4690-abad-332006a3882a" volumeName="kubernetes.io/empty-dir/9f3cf821-229a-4690-abad-332006a3882a-audit-log" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929514 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a2b8eb86-b9d8-4e70-8c4e-736070d94081" volumeName="kubernetes.io/empty-dir/a2b8eb86-b9d8-4e70-8c4e-736070d94081-volume-directive-shadow" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929521 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b8014993-7131-499c-98b3-57b1b7429729" volumeName="kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-oauth-config" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929529 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="be63e086-3851-459b-9123-262570be3b86" volumeName="kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-script-lib" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929537 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cc9755b7-6248-49a7-93c2-06a74060d379" volumeName="kubernetes.io/projected/cc9755b7-6248-49a7-93c2-06a74060d379-kube-api-access-qkbpx" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929545 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" volumeName="kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929552 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a" volumeName="kubernetes.io/empty-dir/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-textfile" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929579 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a" volumeName="kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-tls" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929587 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-tls-assets" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929595 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929620 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a6e33d75-acf8-44b3-a6e3-7599b04fa064" volumeName="kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929628 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cb25da1d-2305-4f75-9920-e52fe596f5b7" volumeName="kubernetes.io/projected/cb25da1d-2305-4f75-9920-e52fe596f5b7-kube-api-access-cvtgq" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929636 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" volumeName="kubernetes.io/projected/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-kube-api-access-vn2zm" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929643 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e22030b7-c861-47ee-b7d7-75f6d659e9c2" volumeName="kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-oauth-config" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929652 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a" volumeName="kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-metrics-client-ca" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929660 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5874f474-b442-4eec-a808-33997e342cb5" volumeName="kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-tmp" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929668 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e15255-91bb-411a-9c47-3c714735e70d" volumeName="kubernetes.io/projected/c8e15255-91bb-411a-9c47-3c714735e70d-kube-api-access-v87r5" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929676 6400 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c8e15255-91bb-411a-9c47-3c714735e70d" volumeName="kubernetes.io/secret/c8e15255-91bb-411a-9c47-3c714735e70d-metrics-tls" seLinuxMountContext="" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929683 6400 reconstruct.go:97] "Volume reconstruction finished" Oct 30 16:21:30.930142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.929687 6400 reconciler.go:29] "Reconciler: start to sync state" Oct 30 16:21:30.944303 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.944250 6400 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 16:21:30.959281 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.959238 6400 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 16:21:30.959281 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.959281 6400 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 16:21:30.959474 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.959303 6400 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 16:21:30.959474 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:30.959347 6400 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 16:21:30.963095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.963073 6400 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 16:21:30.963209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.963173 6400 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:21:30.969942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.969920 6400 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:21:30.970078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.970058 6400 generic.go:356] "Generic (PLEG): container finished" podID="31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a" containerID="90a971174a26fbb50a33127807e1f192126f21e3f65b94a667246c7b94b3339a" exitCode=0 Oct 30 16:21:30.979239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.979212 6400 generic.go:356] "Generic (PLEG): container finished" podID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerID="a2e5530d27d52d0b4ff9b07d782b7281dc740e638b0f0d7d2e3c973d93ca9b26" exitCode=0 Oct 30 16:21:30.981806 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.981786 6400 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:21:30.982176 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.982158 6400 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 16:21:30.994335 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.994303 6400 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="c9d20295d1fbe7cf1d0bfbd1a8cfe0cc183043297b8690d6afbf8623e252cc0b" exitCode=0 Oct 30 16:21:30.994335 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.994336 6400 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="b3c19736369e5e211535094ac95ba764667fe5e2372c442dccb19a8e93167f64" exitCode=0 Oct 30 16:21:30.994504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.994345 6400 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="0b75f64d87246ad1b8a5452226c9838caf0f782eed4d45808ee4b96747624e89" exitCode=0 Oct 30 16:21:30.994504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.994353 6400 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="ba415ee5db4580a713708763a9fb6ca6432a02a653a9b75f3baa81f8d5cb347f" exitCode=0 Oct 30 16:21:30.994504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.994362 6400 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="bb1e64256263c66b4f519130128df8744a7dd9fe28181a078906be19129a7a51" exitCode=0 Oct 30 16:21:30.994504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:30.994370 6400 generic.go:356] "Generic (PLEG): container finished" podID="cf4747b6-b5df-4e8e-a4f7-6de80acaad72" containerID="ee5acd6570feb185531357147ad8c37ea41f1d7043f5e9693048033544c6f9fd" exitCode=0 Oct 30 16:21:31.020593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.020566 6400 generic.go:356] "Generic (PLEG): container finished" podID="c459e3b80b3dfd1e7e8df0c149b8ceb5" containerID="39adf7c56cacde2be481386b6435f0530ef499db9267370a85ddfad6c380ec3f" exitCode=0 Oct 30 16:21:31.037067 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.037045 6400 manager.go:324] Recovery completed Oct 30 16:21:31.059471 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:31.059405 6400 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 16:21:31.060044 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060030 6400 cpu_manager.go:226] "Starting" policy="none" Oct 30 16:21:31.060096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060043 6400 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 16:21:31.060096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060059 6400 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 16:21:31.060186 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060173 6400 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 16:21:31.060215 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060187 6400 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 16:21:31.060246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060216 6400 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 16:21:31.060246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060222 6400 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 16:21:31.060246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060227 6400 policy_none.go:50] "Start" Oct 30 16:21:31.060246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060236 6400 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 16:21:31.060246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060244 6400 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 16:21:31.060371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060361 6400 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 16:21:31.060371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060368 6400 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 16:21:31.060422 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.060373 6400 policy_none.go:44] "Start" Oct 30 16:21:31.064885 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.064858 6400 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 16:21:31.067519 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.067505 6400 manager.go:341] "Starting Device Plugin manager" Oct 30 16:21:31.068019 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:31.067533 6400 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 16:21:31.068019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.068020 6400 server.go:87] "Starting device plugin registration server" Oct 30 16:21:31.068301 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.068288 6400 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 16:21:31.068337 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.068303 6400 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 16:21:31.068414 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.068398 6400 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 16:21:31.068651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.068514 6400 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 16:21:31.068651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.068561 6400 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 16:21:31.068651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.068568 6400 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 16:21:31.068820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.068636 6400 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T16:21:31.068552084Z","UUID":"6f8560ab-251c-448d-a6d6-ecd1da2f1110","Handler":null,"Name":"","Endpoint":""} Oct 30 16:21:31.069844 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.069823 6400 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 16:21:31.069934 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.069855 6400 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 16:21:31.070364 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:31.070343 6400 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 16:21:31.169066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.169033 6400 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 16:21:31.171294 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.171275 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientMemory" Oct 30 16:21:31.171394 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.171307 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 16:21:31.171394 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.171321 6400 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-128-181.ec2.internal" event="NodeHasSufficientPID" Oct 30 16:21:31.171394 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.171368 6400 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.202105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.202085 6400 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.202172 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.202148 6400 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.259994 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.259932 6400 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal"] Oct 30 16:21:31.260653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.260637 6400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.260713 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.260666 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" event={"ID":"3c2020069c5384399725929d6ee8319e","Type":"ContainerStarted","Data":"f0cbbf21b46508a43f980d555347c6ea4ee62a01dd645df91ff681c6de1dc28c"} Oct 30 16:21:31.260754 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.260723 6400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.260812 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.260726 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" event={"ID":"3c2020069c5384399725929d6ee8319e","Type":"ContainerStarted","Data":"6d0ffdfc11a08f5cc3957740a7e3ef60833c74db082289fa4e381f10835cc270"} Oct 30 16:21:31.260880 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.260869 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" event={"ID":"c459e3b80b3dfd1e7e8df0c149b8ceb5","Type":"ContainerStarted","Data":"2a721d5ebbeab1cd2c948704c44167af366ad2c5654b9fffe18419a866a8c583"} Oct 30 16:21:31.260937 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.260887 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" event={"ID":"c459e3b80b3dfd1e7e8df0c149b8ceb5","Type":"ContainerDied","Data":"39adf7c56cacde2be481386b6435f0530ef499db9267370a85ddfad6c380ec3f"} Oct 30 16:21:31.260937 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.260900 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" event={"ID":"c459e3b80b3dfd1e7e8df0c149b8ceb5","Type":"ContainerStarted","Data":"a42f88609e94dc9ee337ffff0f37772fbaef9f59f4cdf4df369a135c2a1bba4c"} Oct 30 16:21:31.282085 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.282063 6400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:21:31.282154 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:31.282110 6400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-181.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.282761 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.282748 6400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:21:31.282862 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:31.282788 6400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.365180 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.365108 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c2020069c5384399725929d6ee8319e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-181.ec2.internal\" (UID: \"3c2020069c5384399725929d6ee8319e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.365180 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.365140 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.365180 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.365159 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.465758 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.465726 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.465758 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.465759 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.465930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.465778 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c2020069c5384399725929d6ee8319e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-181.ec2.internal\" (UID: \"3c2020069c5384399725929d6ee8319e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.465930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.465824 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.465930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.465835 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c459e3b80b3dfd1e7e8df0c149b8ceb5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" (UID: \"c459e3b80b3dfd1e7e8df0c149b8ceb5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.465930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.465871 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/3c2020069c5384399725929d6ee8319e-config\") pod \"kube-apiserver-proxy-ip-10-0-128-181.ec2.internal\" (UID: \"3c2020069c5384399725929d6ee8319e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:21:31.910972 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.910929 6400 apiserver.go:52] "Watching apiserver" Oct 30 16:21:31.959251 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.959218 6400 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 16:21:31.959897 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.959857 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-rns4g","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal","openshift-network-diagnostics/network-check-target-gccnz","kube-system/konnectivity-agent-glrfx","openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb","kube-system/global-pull-secret-syncer-26nhz","openshift-insights/insights-runtime-extractor-6qzkb","openshift-multus/network-metrics-daemon-9ls8t","openshift-ovn-kubernetes/ovnkube-node-w6x6l","openshift-console/console-565777cb7f-d862k","openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57","openshift-console/console-c966567fd-rv9sr","openshift-dns/dns-default-qhm42","openshift-multus/multus-2bm4t","openshift-network-operator/iptables-alerter-p528z","openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr","openshift-monitoring/metrics-server-5f87899b87-ckzpf","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj","openshift-console/console-d8cfd65cf-hkkx4","openshift-monitoring/alertmanager-main-0","kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal","openshift-ingress-canary/ingress-canary-v998h","openshift-monitoring/node-exporter-vgfj7","openshift-multus/multus-additional-cni-plugins-n8mr5","openshift-dns/node-resolver-68psh","openshift-cluster-node-tuning-operator/tuned-772f2"] Oct 30 16:21:31.974053 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.973764 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-jrplp\"" Oct 30 16:21:31.974053 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.973807 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-m64dw\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.975840 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fpzw7\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.975972 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hxhjs\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.976134 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-46f47\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.976303 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.976429 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.976572 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-gfrzf\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.976685 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.976810 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.976934 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lngbh\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.977100 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.977197 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.977310 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.977430 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.977511 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.978262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.977748 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 16:21:31.978959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.978377 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.978959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.978382 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.978959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.978391 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.978959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.978444 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 16:21:31.978959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.978466 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 16:21:31.978959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.978553 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.978959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.978637 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.978959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.978959 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 16:21:31.980663 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.980565 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-sg8tz\"" Oct 30 16:21:31.980663 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.980624 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 16:21:31.980815 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.980738 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.980815 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.980768 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 16:21:31.981238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.981223 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.981974 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.981958 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 16:21:31.982050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.981994 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 16:21:31.982050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.982010 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 16:21:31.982239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.982225 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.982727 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.982707 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 16:21:31.982820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.982746 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.983238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983213 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.983238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983229 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 16:21:31.983357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983253 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 16:21:31.983357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983333 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 16:21:31.983466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983372 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.983466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983422 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 16:21:31.983466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983433 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.983466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983448 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-6q4zp\"" Oct 30 16:21:31.983466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983459 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 16:21:31.983673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983557 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.983673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983522 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.983673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983618 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.983673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983655 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 16:21:31.983836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983691 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 16:21:31.983836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983748 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 16:21:31.983836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983778 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.983836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983794 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-d8rcz\"" Oct 30 16:21:31.983836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.983810 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-5kvnn\"" Oct 30 16:21:31.984050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984032 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-vgrkr\"" Oct 30 16:21:31.984110 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984041 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lbtjk\"" Oct 30 16:21:31.984110 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984058 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 16:21:31.984110 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984099 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-95j9k\"" Oct 30 16:21:31.984273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984128 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 16:21:31.984273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984136 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 16:21:31.984845 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984828 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 16:21:31.984920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984868 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 16:21:31.984920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984877 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 16:21:31.984920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984877 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 16:21:31.984920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984877 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:21:31.984920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.984904 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 16:21:31.985171 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.985155 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 16:21:31.985301 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.985288 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 16:21:31.985902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.985852 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 16:21:31.985902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.985862 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 16:21:31.985902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.985895 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 16:21:31.986079 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986041 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-m9dmv\"" Oct 30 16:21:31.986079 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986059 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cgp6b\"" Oct 30 16:21:31.986172 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986156 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 16:21:31.986223 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986171 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 16:21:31.986270 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986221 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 16:21:31.986270 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986230 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 16:21:31.986270 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986221 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 16:21:31.986409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986272 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-kx92s\"" Oct 30 16:21:31.986409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986284 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.986409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986331 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 16:21:31.986409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986338 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:21:31.986409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986376 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 16:21:31.986409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986385 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 16:21:31.986409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986403 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 16:21:31.986774 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986756 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-s8kck\"" Oct 30 16:21:31.986871 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986814 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-n7jsz\"" Oct 30 16:21:31.986871 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986821 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 16:21:31.986871 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986817 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-r9gmj\"" Oct 30 16:21:31.986871 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986816 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l28xz\"" Oct 30 16:21:31.986871 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986850 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\"" Oct 30 16:21:31.986871 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.986872 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-46k7l\"" Oct 30 16:21:31.990136 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.990122 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-ersnsplrsmf0e\"" Oct 30 16:21:31.994162 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.994147 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 16:21:31.994873 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:31.994859 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 16:21:32.023903 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.023887 6400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:21:32.024015 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.024004 6400 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:21:32.028126 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.028107 6400 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 16:21:32.044922 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.044899 6400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:21:32.044996 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:32.044943 6400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-128-181.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-128-181.ec2.internal" Oct 30 16:21:32.045725 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.045711 6400 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 16:21:32.045780 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:32.045744 6400 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-128-181.ec2.internal" Oct 30 16:21:32.067707 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.067687 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.067803 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.067716 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-socket-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.067803 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.067733 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-registration-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.067803 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.067751 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a2b8eb86-b9d8-4e70-8c4e-736070d94081-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.067803 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.067773 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.068022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.067817 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-ovn\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.068022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.067891 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-log-socket\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.068022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.067913 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/a2b8eb86-b9d8-4e70-8c4e-736070d94081-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.068022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.067956 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-config\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.068022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068002 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:32.068227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068036 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.068227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068037 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-out\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.068227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068072 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.068227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068089 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.068227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068113 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8014993-7131-499c-98b3-57b1b7429729-tmp\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.068227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068122 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-out\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.068227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068136 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.068227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068165 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.068227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068186 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-config\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068235 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8014993-7131-499c-98b3-57b1b7429729-tmp\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068243 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068304 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-node-log\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068317 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068360 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068378 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-netns\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068413 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-kubelet\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068423 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-config\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068442 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/da3c94aa-c880-4844-8cc8-2182ff86e321-original-pull-secret\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068464 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-oauth-config\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068500 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068545 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068572 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-web-config\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068620 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068646 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.068683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068685 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-oauth-config\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068701 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-netd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068715 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068729 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-system-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068699 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-config\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068765 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068799 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-multus-certs\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068810 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-web-config\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068827 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-kubelet-config\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068863 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068894 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-crio-socket\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068908 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068923 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ds4gb\" (UniqueName: \"kubernetes.io/projected/be63e086-3851-459b-9123-262570be3b86-kube-api-access-ds4gb\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068959 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-bin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068981 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.068987 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069013 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v87r5\" (UniqueName: \"kubernetes.io/projected/c8e15255-91bb-411a-9c47-3c714735e70d-kube-api-access-v87r5\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069042 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069067 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e50ac2ed-9597-47ec-a4be-078d412f0b61-iptables-alerter-script\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069081 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-volume\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069097 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-hostroot\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069128 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwsxf\" (UniqueName: \"kubernetes.io/projected/72976414-f225-493b-9c55-3ab89bb831a8-kube-api-access-bwsxf\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069156 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-systemd\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069184 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cnibin\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069221 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069256 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8e15255-91bb-411a-9c47-3c714735e70d-metrics-tls\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069272 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/e50ac2ed-9597-47ec-a4be-078d412f0b61-iptables-alerter-script\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069277 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069293 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-etc-tuned\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069338 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.069357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069365 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069390 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-service-ca\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069417 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12134b5c-de72-4972-a185-3d5450c7b5e7-agent-certs\") pod \"konnectivity-agent-glrfx\" (UID: \"12134b5c-de72-4972-a185-3d5450c7b5e7\") " pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069421 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-etc-tuned\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069439 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-tmp\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069458 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c8e15255-91bb-411a-9c47-3c714735e70d-metrics-tls\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069463 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069500 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-lib-modules\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069516 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-systemd-units\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069550 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069554 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-tmp\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069563 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069581 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-var-lib-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069595 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-service-ca\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069642 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-kubelet\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069690 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-textfile\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069718 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysconfig\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069720 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/12134b5c-de72-4972-a185-3d5450c7b5e7-agent-certs\") pod \"konnectivity-agent-glrfx\" (UID: \"12134b5c-de72-4972-a185-3d5450c7b5e7\") " pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069762 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/569992a0-aa75-48cb-ad8d-fb61295a5093-metrics-certs\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069759 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-textfile\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069786 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069812 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwlmr\" (UniqueName: \"kubernetes.io/projected/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-api-access-hwlmr\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069828 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069848 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-serving-cert\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069873 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-multus\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.069939 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070215 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-conf-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070274 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqjj\" (UniqueName: \"kubernetes.io/projected/45b113ed-a156-4c53-bee5-26707e5717c3-kube-api-access-5hqjj\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070302 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070327 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-serving-cert\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070367 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070449 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-serving-cert\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070521 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-service-ca\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.070548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070562 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-system-cni-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070576 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a2b8eb86-b9d8-4e70-8c4e-736070d94081-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070588 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070657 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-serving-cert\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070824 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-service-ca\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070595 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-client-certs\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070873 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f3cf821-229a-4690-abad-332006a3882a-audit-log\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070937 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-client-certs\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070945 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/9f3cf821-229a-4690-abad-332006a3882a-audit-log\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070947 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgv82\" (UniqueName: \"kubernetes.io/projected/e50ac2ed-9597-47ec-a4be-078d412f0b61-kube-api-access-sgv82\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.070986 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-tls\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071021 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e15255-91bb-411a-9c47-3c714735e70d-config-volume\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071111 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-sys\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071148 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-slash\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071183 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cns2t\" (UniqueName: \"kubernetes.io/projected/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-kube-api-access-cns2t\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071207 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-run\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071218 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c8e15255-91bb-411a-9c47-3c714735e70d-config-volume\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071238 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071241 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-tls\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071294 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-metrics-server-audit-profiles\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071444 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-trusted-ca-bundle\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071480 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-dbus\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071522 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw7vq\" (UniqueName: \"kubernetes.io/projected/5874f474-b442-4eec-a808-33997e342cb5-kube-api-access-lw7vq\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071548 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071692 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-netns\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071742 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071794 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-sys\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071833 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn2zm\" (UniqueName: \"kubernetes.io/projected/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-kube-api-access-vn2zm\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071886 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kmlq\" (UniqueName: \"kubernetes.io/projected/b8014993-7131-499c-98b3-57b1b7429729-kube-api-access-5kmlq\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.071987 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-trusted-ca-bundle\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072054 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-multus-daemon-config\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.072076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072083 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2dv7\" (UniqueName: \"kubernetes.io/projected/e22030b7-c861-47ee-b7d7-75f6d659e9c2-kube-api-access-j2dv7\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072119 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-serving-cert\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072174 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6gbn\" (UniqueName: \"kubernetes.io/projected/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-kube-api-access-z6gbn\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072204 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvtgq\" (UniqueName: \"kubernetes.io/projected/cb25da1d-2305-4f75-9920-e52fe596f5b7-kube-api-access-cvtgq\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072229 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-kubernetes\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072259 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-script-lib\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072289 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-etc-kubernetes\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072432 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072478 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-host\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072533 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072563 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-ovnkube-script-lib\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072581 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072641 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-cnibin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072670 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-k8s-cni-cncf-io\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072719 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-root\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072757 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-wtmp\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072823 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072874 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb25da1d-2305-4f75-9920-e52fe596f5b7-hosts-file\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072875 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/9f3cf821-229a-4690-abad-332006a3882a-metrics-server-audit-profiles\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072937 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-kube-api-access-9cbmh\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.072973 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-env-overrides\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073000 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073031 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12134b5c-de72-4972-a185-3d5450c7b5e7-konnectivity-ca\") pod \"konnectivity-agent-glrfx\" (UID: \"12134b5c-de72-4972-a185-3d5450c7b5e7\") " pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073061 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72976414-f225-493b-9c55-3ab89bb831a8-serviceca\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073138 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-multus-daemon-config\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073162 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-var-lib-kubelet\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073195 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073226 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e50ac2ed-9597-47ec-a4be-078d412f0b61-host-slash\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073256 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-sys-fs\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073290 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flf2g\" (UniqueName: \"kubernetes.io/projected/9b24f224-6dab-4478-890b-9da0d0679187-kube-api-access-flf2g\") pod \"monitoring-plugin-77f4f76594-hlrgr\" (UID: \"9b24f224-6dab-4478-890b-9da0d0679187\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073321 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-conf\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073351 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-oauth-serving-cert\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073381 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-socket-dir-parent\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073411 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-device-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.073522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.073442 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-config\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074026 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-serving-cert\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074412 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074452 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqrzv\" (UniqueName: \"kubernetes.io/projected/9f3cf821-229a-4690-abad-332006a3882a-kube-api-access-rqrzv\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074486 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmztd\" (UniqueName: \"kubernetes.io/projected/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-kube-api-access-qmztd\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074515 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-metrics-client-ca\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074547 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074577 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72976414-f225-493b-9c55-3ab89bb831a8-host\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074636 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-trusted-ca-bundle\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074665 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074698 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-cni-binary-copy\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074727 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22030b7-c861-47ee-b7d7-75f6d659e9c2-tmp\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074805 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25da1d-2305-4f75-9920-e52fe596f5b7-tmp-dir\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074837 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwrf\" (UniqueName: \"kubernetes.io/projected/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-kube-api-access-ldwrf\") pod \"ingress-canary-v998h\" (UID: \"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a\") " pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074863 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.075034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.074894 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e15255-91bb-411a-9c47-3c714735e70d-tmp-dir\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:32.075692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.075101 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22030b7-c861-47ee-b7d7-75f6d659e9c2-tmp\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.075692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.075183 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-oauth-serving-cert\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.075692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.075213 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:32.075692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.075266 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.075692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.075304 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-metrics-client-ca\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.075692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.075323 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cni-binary-copy\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.075692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.075476 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-config\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.075692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.075508 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/be63e086-3851-459b-9123-262570be3b86-env-overrides\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.075958 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.075751 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-tls-assets\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.076510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076160 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c8e15255-91bb-411a-9c47-3c714735e70d-tmp-dir\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:32.076510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076166 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-modprobe-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.076510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076187 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/12134b5c-de72-4972-a185-3d5450c7b5e7-konnectivity-ca\") pod \"konnectivity-agent-glrfx\" (UID: \"12134b5c-de72-4972-a185-3d5450c7b5e7\") " pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:21:32.076510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076216 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x66t\" (UniqueName: \"kubernetes.io/projected/569992a0-aa75-48cb-ad8d-fb61295a5093-kube-api-access-4x66t\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:32.076510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076221 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/cb25da1d-2305-4f75-9920-e52fe596f5b7-tmp-dir\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:21:32.076510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076328 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-oauth-serving-cert\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.076510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076383 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.076510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076395 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-oauth-config\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.076510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076482 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-oauth-serving-cert\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.076510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076492 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/72976414-f225-493b-9c55-3ab89bb831a8-serviceca\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076529 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076562 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-oauth-config\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076572 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-oauth-config\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076644 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-tmp\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076720 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/be63e086-3851-459b-9123-262570be3b86-ovn-node-metrics-cert\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076733 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5874f474-b442-4eec-a808-33997e342cb5-tmp\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076791 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gnnr\" (UniqueName: \"kubernetes.io/projected/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-api-access-2gnnr\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076816 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-oauth-config\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076821 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-service-ca\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076836 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/45b113ed-a156-4c53-bee5-26707e5717c3-cni-binary-copy\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076879 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-bin\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.076943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076918 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-trusted-ca-bundle\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.076973 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-service-ca\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077009 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-data-volume\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077037 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/be63e086-3851-459b-9123-262570be3b86-ovn-node-metrics-cert\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077046 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-tls\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077094 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-data-volume\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077093 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-etc-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077126 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-client-ca-bundle\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077162 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-oauth-serving-cert\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077207 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-secret-metrics-server-tls\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077258 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077278 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qkbpx\" (UniqueName: \"kubernetes.io/projected/cc9755b7-6248-49a7-93c2-06a74060d379-kube-api-access-qkbpx\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077309 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-cert\") pod \"ingress-canary-v998h\" (UID: \"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a\") " pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:32.077418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077364 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/9f3cf821-229a-4690-abad-332006a3882a-client-ca-bundle\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.077874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077457 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-oauth-serving-cert\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.077874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077465 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:32.077874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077460 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-cert\") pod \"ingress-canary-v998h\" (UID: \"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a\") " pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:32.077874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077477 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.077874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077516 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-console-config\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.077874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077534 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b24f224-6dab-4478-890b-9da0d0679187-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hlrgr\" (UID: \"9b24f224-6dab-4478-890b-9da0d0679187\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:32.077874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077552 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-systemd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.077874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077617 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.077874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077781 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9b24f224-6dab-4478-890b-9da0d0679187-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-hlrgr\" (UID: \"9b24f224-6dab-4478-890b-9da0d0679187\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:32.077874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.077848 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 16:21:32.078160 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.078145 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-console-config\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.078287 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.078275 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-accelerators-collector-config\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.089226 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.089210 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ds4gb\" (UniqueName: \"kubernetes.io/projected/be63e086-3851-459b-9123-262570be3b86-kube-api-access-ds4gb\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.089643 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.089629 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwsxf\" (UniqueName: \"kubernetes.io/projected/72976414-f225-493b-9c55-3ab89bb831a8-kube-api-access-bwsxf\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:21:32.090555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.090540 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwlmr\" (UniqueName: \"kubernetes.io/projected/a2b8eb86-b9d8-4e70-8c4e-736070d94081-kube-api-access-hwlmr\") pod \"kube-state-metrics-5dc8d7d89b-ctcjb\" (UID: \"a2b8eb86-b9d8-4e70-8c4e-736070d94081\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-ctcjb" Oct 30 16:21:32.093660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.093633 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqjj\" (UniqueName: \"kubernetes.io/projected/45b113ed-a156-4c53-bee5-26707e5717c3-kube-api-access-5hqjj\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.096403 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.096381 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgv82\" (UniqueName: \"kubernetes.io/projected/e50ac2ed-9597-47ec-a4be-078d412f0b61-kube-api-access-sgv82\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:21:32.096507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.096489 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwrf\" (UniqueName: \"kubernetes.io/projected/0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a-kube-api-access-ldwrf\") pod \"ingress-canary-v998h\" (UID: \"0dd1218c-9c3d-4d82-81f2-3cf601ebeb9a\") " pod="openshift-ingress-canary/ingress-canary-v998h" Oct 30 16:21:32.096994 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.096979 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kc9p\" (UniqueName: \"kubernetes.io/projected/67d3a22c-930e-4113-ba61-2aecccabf9d2-kube-api-access-8kc9p\") pod \"network-check-target-gccnz\" (UID: \"67d3a22c-930e-4113-ba61-2aecccabf9d2\") " pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:32.097708 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.097692 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flf2g\" (UniqueName: \"kubernetes.io/projected/9b24f224-6dab-4478-890b-9da0d0679187-kube-api-access-flf2g\") pod \"monitoring-plugin-77f4f76594-hlrgr\" (UID: \"9b24f224-6dab-4478-890b-9da0d0679187\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:32.098660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.098641 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw7vq\" (UniqueName: \"kubernetes.io/projected/5874f474-b442-4eec-a808-33997e342cb5-kube-api-access-lw7vq\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.100251 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.100231 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6gbn\" (UniqueName: \"kubernetes.io/projected/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-kube-api-access-z6gbn\") pod \"console-565777cb7f-d862k\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:32.100966 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.100947 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cns2t\" (UniqueName: \"kubernetes.io/projected/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-kube-api-access-cns2t\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.101038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.100994 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn2zm\" (UniqueName: \"kubernetes.io/projected/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-kube-api-access-vn2zm\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.102056 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.102039 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2dv7\" (UniqueName: \"kubernetes.io/projected/e22030b7-c861-47ee-b7d7-75f6d659e9c2-kube-api-access-j2dv7\") pod \"console-c966567fd-rv9sr\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:32.107444 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.107420 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x66t\" (UniqueName: \"kubernetes.io/projected/569992a0-aa75-48cb-ad8d-fb61295a5093-kube-api-access-4x66t\") pod \"network-metrics-daemon-9ls8t\" (UID: \"569992a0-aa75-48cb-ad8d-fb61295a5093\") " pod="openshift-multus/network-metrics-daemon-9ls8t" Oct 30 16:21:32.107545 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.107529 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kmlq\" (UniqueName: \"kubernetes.io/projected/b8014993-7131-499c-98b3-57b1b7429729-kube-api-access-5kmlq\") pod \"console-d8cfd65cf-hkkx4\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:32.107595 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.107569 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvtgq\" (UniqueName: \"kubernetes.io/projected/cb25da1d-2305-4f75-9920-e52fe596f5b7-kube-api-access-cvtgq\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:21:32.107595 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.107572 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-kube-api-access-9cbmh\") pod \"alertmanager-main-0\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:32.107947 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.107923 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmztd\" (UniqueName: \"kubernetes.io/projected/e01c7dab-d74b-4efe-9a0b-58a9eba6d956-kube-api-access-qmztd\") pod \"openshift-state-metrics-7bd7489957-5sc57\" (UID: \"e01c7dab-d74b-4efe-9a0b-58a9eba6d956\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-5sc57" Oct 30 16:21:32.108238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.108206 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gnnr\" (UniqueName: \"kubernetes.io/projected/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-kube-api-access-2gnnr\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.108472 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.108458 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqrzv\" (UniqueName: \"kubernetes.io/projected/9f3cf821-229a-4690-abad-332006a3882a-kube-api-access-rqrzv\") pod \"metrics-server-5f87899b87-ckzpf\" (UID: \"9f3cf821-229a-4690-abad-332006a3882a\") " pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:32.113758 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.113733 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qkbpx\" (UniqueName: \"kubernetes.io/projected/cc9755b7-6248-49a7-93c2-06a74060d379-kube-api-access-qkbpx\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.115302 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.115280 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v87r5\" (UniqueName: \"kubernetes.io/projected/c8e15255-91bb-411a-9c47-3c714735e70d-kube-api-access-v87r5\") pod \"dns-default-qhm42\" (UID: \"c8e15255-91bb-411a-9c47-3c714735e70d\") " pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:32.150195 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.150167 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c966567fd-rv9sr"] Oct 30 16:21:32.178696 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178628 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-cnibin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.178696 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178657 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-k8s-cni-cncf-io\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.178696 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178676 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-root\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178701 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-wtmp\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178734 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178740 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-k8s-cni-cncf-io\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178756 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb25da1d-2305-4f75-9920-e52fe596f5b7-hosts-file\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178745 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-cnibin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178788 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-etc-selinux\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178807 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/cb25da1d-2305-4f75-9920-e52fe596f5b7-hosts-file\") pod \"node-resolver-68psh\" (UID: \"cb25da1d-2305-4f75-9920-e52fe596f5b7\") " pod="openshift-dns/node-resolver-68psh" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178787 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-var-lib-kubelet\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178788 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-node-exporter-wtmp\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178820 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-var-lib-kubelet\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178836 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e50ac2ed-9597-47ec-a4be-078d412f0b61-host-slash\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178853 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-sys-fs\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178747 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-root\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178874 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-conf\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178900 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-socket-dir-parent\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178908 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e50ac2ed-9597-47ec-a4be-078d412f0b61-host-slash\") pod \"iptables-alerter-p528z\" (UID: \"e50ac2ed-9597-47ec-a4be-078d412f0b61\") " pod="openshift-network-operator/iptables-alerter-p528z" Oct 30 16:21:32.178919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178922 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-device-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178941 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-socket-dir-parent\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178914 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-sys-fs\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178953 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178967 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-device-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178977 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72976414-f225-493b-9c55-3ab89bb831a8-host\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.178994 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-conf\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179000 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179006 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-kubelet-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179024 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-modprobe-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179035 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/72976414-f225-493b-9c55-3ab89bb831a8-host\") pod \"node-ca-rns4g\" (UID: \"72976414-f225-493b-9c55-3ab89bb831a8\") " pod="openshift-image-registry/node-ca-rns4g" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179047 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-bin\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179065 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-etc-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179070 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179086 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-modprobe-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179086 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-systemd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179112 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-socket-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179122 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-etc-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179127 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-registration-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179142 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-socket-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179146 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-ovn\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179111 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-systemd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179153 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/cc9755b7-6248-49a7-93c2-06a74060d379-registration-dir\") pod \"aws-ebs-csi-driver-node-xhkrj\" (UID: \"cc9755b7-6248-49a7-93c2-06a74060d379\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-xhkrj" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179123 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-bin\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179161 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-log-socket\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179176 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-ovn\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179181 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-log-socket\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179190 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179206 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179219 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-node-log\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179241 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179261 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-run-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179274 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-node-log\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179261 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-netns\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179289 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-ovn-kubernetes\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179294 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-netns\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179320 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179318 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-kubelet\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179350 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-netd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179352 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-kubelet\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179365 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-system-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179384 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-cni-netd\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179389 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-multus-certs\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179413 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-system-cni-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179416 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-kubelet-config\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179439 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-crio-socket\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179453 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-bin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179476 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-hostroot\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179488 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-kubelet-config\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179498 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-hostroot\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179502 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-systemd\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179491 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-run-multus-certs\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179496 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-bin\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179479 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/d1dc93b8-6bd0-484e-a06c-d5a659e44eba-crio-socket\") pod \"insights-runtime-extractor-6qzkb\" (UID: \"d1dc93b8-6bd0-484e-a06c-d5a659e44eba\") " pod="openshift-insights/insights-runtime-extractor-6qzkb" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179531 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cnibin\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179542 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-systemd\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.179525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179565 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179562 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-cnibin\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179589 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-lib-modules\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179632 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-systemd-units\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179656 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-var-lib-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179677 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-systemd-units\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179681 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-kubelet\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179705 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-lib-modules\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179710 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysconfig\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179713 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-kubelet\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179705 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysctl-d\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179737 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-var-lib-openvswitch\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179749 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-sysconfig\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179757 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-multus\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179777 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-conf-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179795 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-system-cni-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179811 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-sys\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179829 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-slash\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179845 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-host-var-lib-cni-multus\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179849 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-system-cni-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179852 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-run\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179865 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-sys\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179874 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-multus-conf-dir\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179884 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-dbus\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179886 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-run\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179887 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-slash\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179923 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/da3c94aa-c880-4844-8cc8-2182ff86e321-dbus\") pod \"global-pull-secret-syncer-26nhz\" (UID: \"da3c94aa-c880-4844-8cc8-2182ff86e321\") " pod="kube-system/global-pull-secret-syncer-26nhz" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179941 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-netns\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179960 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179973 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/be63e086-3851-459b-9123-262570be3b86-host-run-netns\") pod \"ovnkube-node-w6x6l\" (UID: \"be63e086-3851-459b-9123-262570be3b86\") " pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179973 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-sys\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.179997 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a-sys\") pod \"node-exporter-vgfj7\" (UID: \"31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a\") " pod="openshift-monitoring/node-exporter-vgfj7" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.180010 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-kubernetes\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.180026 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-etc-kubernetes\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.180035 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf4747b6-b5df-4e8e-a4f7-6de80acaad72-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n8mr5\" (UID: \"cf4747b6-b5df-4e8e-a4f7-6de80acaad72\") " pod="openshift-multus/multus-additional-cni-plugins-n8mr5" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.180071 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-host\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.180077 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-etc-kubernetes\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.180041 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5874f474-b442-4eec-a808-33997e342cb5-host\") pod \"tuned-772f2\" (UID: \"5874f474-b442-4eec-a808-33997e342cb5\") " pod="openshift-cluster-node-tuning-operator/tuned-772f2" Oct 30 16:21:32.181406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:32.180100 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/45b113ed-a156-4c53-bee5-26707e5717c3-etc-kubernetes\") pod \"multus-2bm4t\" (UID: \"45b113ed-a156-4c53-bee5-26707e5717c3\") " pod="openshift-multus/multus-2bm4t" Oct 30 16:21:33.933330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:33.933297 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qhm42" Oct 30 16:21:34.070004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:34.069976 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:34.201086 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:34.201013 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:34.485828 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:34.485797 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-w6x6l" Oct 30 16:21:34.500004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:34.499978 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:35.323837 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:35.323801 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:21:37.956359 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:37.956319 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-hlrgr" Oct 30 16:21:39.159985 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:39.159946 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:39.164893 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:39.164869 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:40.234088 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:40.234061 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-5f87899b87-ckzpf" Oct 30 16:21:41.417625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.417572 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-57cc7b5857-4p6bm"] Oct 30 16:21:41.423880 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.423856 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.440273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.437158 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:41.449182 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.449153 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57cc7b5857-4p6bm"] Oct 30 16:21:41.559896 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.559863 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-console-serving-cert\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.559896 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.559894 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-console-config\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.560105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.559914 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-service-ca\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.560105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.560068 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-trusted-ca-bundle\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.560204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.560121 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6h7\" (UniqueName: \"kubernetes.io/projected/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-kube-api-access-ss6h7\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.560265 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.560203 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-console-oauth-config\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.560265 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.560224 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-tmp\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.560364 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.560311 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-oauth-serving-cert\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.661193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.661158 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-console-serving-cert\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.661193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.661196 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-console-config\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.661445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.661216 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-service-ca\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.661445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.661253 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-trusted-ca-bundle\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.661445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.661277 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6h7\" (UniqueName: \"kubernetes.io/projected/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-kube-api-access-ss6h7\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.661445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.661298 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-console-oauth-config\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.661445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.661318 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-tmp\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.661445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.661354 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-oauth-serving-cert\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.661821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.661793 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-tmp\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.661888 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.661823 6400 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 16:21:41.662187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.662161 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-service-ca\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.662187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.662179 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-console-config\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.662637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.662171 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-oauth-serving-cert\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.662637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.662418 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-trusted-ca-bundle\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.664222 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.664194 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-console-oauth-config\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.664315 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.664199 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-console-serving-cert\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.682509 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.682457 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6h7\" (UniqueName: \"kubernetes.io/projected/a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37-kube-api-access-ss6h7\") pod \"console-57cc7b5857-4p6bm\" (UID: \"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37\") " pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:41.736013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:41.735975 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:42.032458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:42.032419 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-57cc7b5857-4p6bm"] Oct 30 16:21:42.036058 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:42.036025 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2a21dfa_119a_4ebb_a58c_09b5e3cb3d37.slice/crio-04a5c5fbda007e57723465c22f5126c69e2fd002b96ea12e3d2fa8e1c96f2b16 WatchSource:0}: Error finding container 04a5c5fbda007e57723465c22f5126c69e2fd002b96ea12e3d2fa8e1c96f2b16: Status 404 returned error can't find the container with id 04a5c5fbda007e57723465c22f5126c69e2fd002b96ea12e3d2fa8e1c96f2b16 Oct 30 16:21:42.061775 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:42.061739 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57cc7b5857-4p6bm" event={"ID":"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37","Type":"ContainerStarted","Data":"04a5c5fbda007e57723465c22f5126c69e2fd002b96ea12e3d2fa8e1c96f2b16"} Oct 30 16:21:43.066898 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:43.066861 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-57cc7b5857-4p6bm" event={"ID":"a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37","Type":"ContainerStarted","Data":"9d015f7c98a147c7833a46bd52c70a03524d4b271459114731dd693afe42872b"} Oct 30 16:21:43.105506 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:43.105441 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-57cc7b5857-4p6bm" podStartSLOduration=2.105421688 podStartE2EDuration="2.105421688s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:43.10497994 +0000 UTC m=+12.255260930" watchObservedRunningTime="2026-10-30 16:21:43.105421688 +0000 UTC m=+12.255702675" Oct 30 16:21:45.828175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:45.828139 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:21:45.828636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:45.828438 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="alertmanager" containerID="cri-o://52a15ac6f58e4511905538277767d83c9d89e40a552b1dcdcad33de6f837429f" gracePeriod=120 Oct 30 16:21:45.828636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:45.828485 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy-metric" containerID="cri-o://dcceb272812eb79c1bbd2ab10113ad8ea84053a65ab531aefe91c7b2a8977add" gracePeriod=120 Oct 30 16:21:45.828636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:45.828546 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="prom-label-proxy" containerID="cri-o://287dd45d55fd2d5aea6a8ccb2844101819747e8229812cbb8afb1ddd7d3cdd87" gracePeriod=120 Oct 30 16:21:45.828636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:45.828588 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy" containerID="cri-o://16fdba6dda4782e29fc5b9866a5d9e996ff8737fb6b47d979212a7bb40e79415" gracePeriod=120 Oct 30 16:21:45.828991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:45.828574 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="config-reloader" containerID="cri-o://8b9e0a723a49c534b112e97c83f0933c27c9b33af7e05276efdd10acce8b300e" gracePeriod=120 Oct 30 16:21:45.828991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:45.828552 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy-web" containerID="cri-o://3b12bba989d729d6102b23b4ab112389882af4ea2af10d31b16ac16790b89ed9" gracePeriod=120 Oct 30 16:21:46.085582 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085518 6400 generic.go:356] "Generic (PLEG): container finished" podID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerID="287dd45d55fd2d5aea6a8ccb2844101819747e8229812cbb8afb1ddd7d3cdd87" exitCode=0 Oct 30 16:21:46.085582 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085541 6400 generic.go:356] "Generic (PLEG): container finished" podID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerID="dcceb272812eb79c1bbd2ab10113ad8ea84053a65ab531aefe91c7b2a8977add" exitCode=0 Oct 30 16:21:46.085582 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085548 6400 generic.go:356] "Generic (PLEG): container finished" podID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerID="16fdba6dda4782e29fc5b9866a5d9e996ff8737fb6b47d979212a7bb40e79415" exitCode=0 Oct 30 16:21:46.085582 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085556 6400 generic.go:356] "Generic (PLEG): container finished" podID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerID="3b12bba989d729d6102b23b4ab112389882af4ea2af10d31b16ac16790b89ed9" exitCode=0 Oct 30 16:21:46.085582 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085561 6400 generic.go:356] "Generic (PLEG): container finished" podID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerID="8b9e0a723a49c534b112e97c83f0933c27c9b33af7e05276efdd10acce8b300e" exitCode=0 Oct 30 16:21:46.085582 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085567 6400 generic.go:356] "Generic (PLEG): container finished" podID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerID="52a15ac6f58e4511905538277767d83c9d89e40a552b1dcdcad33de6f837429f" exitCode=0 Oct 30 16:21:46.085872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085592 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerDied","Data":"287dd45d55fd2d5aea6a8ccb2844101819747e8229812cbb8afb1ddd7d3cdd87"} Oct 30 16:21:46.085872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085680 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerDied","Data":"dcceb272812eb79c1bbd2ab10113ad8ea84053a65ab531aefe91c7b2a8977add"} Oct 30 16:21:46.085872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085709 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerDied","Data":"16fdba6dda4782e29fc5b9866a5d9e996ff8737fb6b47d979212a7bb40e79415"} Oct 30 16:21:46.085872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085734 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerDied","Data":"3b12bba989d729d6102b23b4ab112389882af4ea2af10d31b16ac16790b89ed9"} Oct 30 16:21:46.085872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085755 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerDied","Data":"8b9e0a723a49c534b112e97c83f0933c27c9b33af7e05276efdd10acce8b300e"} Oct 30 16:21:46.085872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085777 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerDied","Data":"52a15ac6f58e4511905538277767d83c9d89e40a552b1dcdcad33de6f837429f"} Oct 30 16:21:46.085872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085799 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"a6e33d75-acf8-44b3-a6e3-7599b04fa064","Type":"ContainerDied","Data":"371dd94fe0177fcf78a797a514dd4cbfba3ccc8d00c215f2e90734fbd5df3ad9"} Oct 30 16:21:46.085872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.085821 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="371dd94fe0177fcf78a797a514dd4cbfba3ccc8d00c215f2e90734fbd5df3ad9" Oct 30 16:21:46.108516 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.108493 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:46.302690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.302659 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-cluster-tls-config\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.302690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.302699 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-main-db\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.302948 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.302722 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-web\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.302948 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.302789 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-tls-assets\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-tls-assets\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.302948 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.302825 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-web-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-web-config\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.302948 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.302848 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-out\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.302948 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.302874 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.302948 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.302921 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-trusted-ca-bundle\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.303286 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.302962 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-metric\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.303286 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.302994 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-metrics-client-ca\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.303286 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.303029 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-main-tls\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.303286 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.303061 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-kube-api-access-9cbmh\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.303286 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.303075 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-main-db" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:21:46.303286 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.303103 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-volume\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-volume\") pod \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\" (UID: \"a6e33d75-acf8-44b3-a6e3-7599b04fa064\") " Oct 30 16:21:46.303787 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.303534 6400 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-main-db\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.305441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.305410 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-web" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:46.305857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.305831 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-trusted-ca-bundle" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:46.306190 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.306082 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-metric" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:46.306190 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.306087 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-tls-assets" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:46.306190 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.306140 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-volume" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:46.306634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.306508 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-metrics-client-ca" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:46.306940 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.306916 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-out" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:21:46.307138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.307116 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:46.308023 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.308005 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-main-tls" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:46.308206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.308192 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-kube-api-access-9cbmh" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "kube-api-access-9cbmh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:46.310316 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.310296 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-cluster-tls-config" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:46.316576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.316558 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-web-config" pod "a6e33d75-acf8-44b3-a6e3-7599b04fa064" (UID: "a6e33d75-acf8-44b3-a6e3-7599b04fa064"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:46.404206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404128 6400 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-web-config\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404158 6400 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-out\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404169 6400 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404179 6400 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404188 6400 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404198 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a6e33d75-acf8-44b3-a6e3-7599b04fa064-metrics-client-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404206 6400 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-main-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404527 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404215 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cbmh\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-kube-api-access-9cbmh\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404527 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404223 6400 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-config-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404527 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404232 6400 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-cluster-tls-config\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404527 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404240 6400 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a6e33d75-acf8-44b3-a6e3-7599b04fa064-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:46.404527 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:46.404249 6400 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a6e33d75-acf8-44b3-a6e3-7599b04fa064-tls-assets\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:47.089442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.089408 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.147189 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.147105 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:21:47.157241 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.157221 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:21:47.190369 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190328 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:21:47.190636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190617 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="config-reloader" Oct 30 16:21:47.190636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190630 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="config-reloader" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190647 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="prom-label-proxy" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190652 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="prom-label-proxy" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190658 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="init-config-reloader" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190663 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="init-config-reloader" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190672 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="alertmanager" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190677 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="alertmanager" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190684 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy-metric" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190689 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy-metric" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190699 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190706 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190731 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy-web" Oct 30 16:21:47.190765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190739 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy-web" Oct 30 16:21:47.191224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190794 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy-web" Oct 30 16:21:47.191224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190806 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="config-reloader" Oct 30 16:21:47.191224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190816 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="prom-label-proxy" Oct 30 16:21:47.191224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190825 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="init-config-reloader" Oct 30 16:21:47.191224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190833 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="alertmanager" Oct 30 16:21:47.191224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190841 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy-metric" Oct 30 16:21:47.191224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.190848 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" containerName="kube-rbac-proxy" Oct 30 16:21:47.224667 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.224637 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:21:47.224778 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.224735 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.238504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.238482 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 16:21:47.238628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.238506 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 16:21:47.238628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.238521 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 16:21:47.238628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.238530 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 16:21:47.238628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.238526 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 16:21:47.238628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.238482 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 16:21:47.238628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.238482 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 16:21:47.238628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.238532 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 16:21:47.238925 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.238752 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-cgp6b\"" Oct 30 16:21:47.248952 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.248933 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 16:21:47.312171 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312142 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-config-volume\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312315 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312181 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d90c80ab-29ef-4083-bd11-74e4ba0c297b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312315 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312199 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312284 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d90c80ab-29ef-4083-bd11-74e4ba0c297b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312336 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90c80ab-29ef-4083-bd11-74e4ba0c297b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312358 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-web-config\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312376 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312588 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312456 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d90c80ab-29ef-4083-bd11-74e4ba0c297b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312588 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312493 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312583 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312627 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h76b7\" (UniqueName: \"kubernetes.io/projected/d90c80ab-29ef-4083-bd11-74e4ba0c297b-kube-api-access-h76b7\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312777 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312701 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d90c80ab-29ef-4083-bd11-74e4ba0c297b-config-out\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.312777 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.312731 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413360 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413260 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413360 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413303 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h76b7\" (UniqueName: \"kubernetes.io/projected/d90c80ab-29ef-4083-bd11-74e4ba0c297b-kube-api-access-h76b7\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413360 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413341 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d90c80ab-29ef-4083-bd11-74e4ba0c297b-config-out\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413375 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413400 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-config-volume\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413430 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d90c80ab-29ef-4083-bd11-74e4ba0c297b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413454 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413484 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d90c80ab-29ef-4083-bd11-74e4ba0c297b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413504 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90c80ab-29ef-4083-bd11-74e4ba0c297b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413534 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-web-config\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413563 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413592 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d90c80ab-29ef-4083-bd11-74e4ba0c297b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.413645 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.415350 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.415321 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d90c80ab-29ef-4083-bd11-74e4ba0c297b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.415779 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.415754 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/d90c80ab-29ef-4083-bd11-74e4ba0c297b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.417843 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.416986 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.417843 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.417453 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.418034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.418007 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d90c80ab-29ef-4083-bd11-74e4ba0c297b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.418437 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.418411 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/d90c80ab-29ef-4083-bd11-74e4ba0c297b-config-out\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.418591 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.418563 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-web-config\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.418737 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.418570 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.418872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.418852 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.418929 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.418877 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.419118 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.419102 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/d90c80ab-29ef-4083-bd11-74e4ba0c297b-config-volume\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.419985 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.419970 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/d90c80ab-29ef-4083-bd11-74e4ba0c297b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.436042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.436020 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h76b7\" (UniqueName: \"kubernetes.io/projected/d90c80ab-29ef-4083-bd11-74e4ba0c297b-kube-api-access-h76b7\") pod \"alertmanager-main-0\" (UID: \"d90c80ab-29ef-4083-bd11-74e4ba0c297b\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.631915 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.631884 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 16:21:47.987111 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:47.987075 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 16:21:47.989917 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:21:47.989890 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90c80ab_29ef_4083_bd11_74e4ba0c297b.slice/crio-d91b1c2c89a6060cf2a20660dd2ea0761cde8150fe1b8befc226d93f98b1b8b2 WatchSource:0}: Error finding container d91b1c2c89a6060cf2a20660dd2ea0761cde8150fe1b8befc226d93f98b1b8b2: Status 404 returned error can't find the container with id d91b1c2c89a6060cf2a20660dd2ea0761cde8150fe1b8befc226d93f98b1b8b2 Oct 30 16:21:48.094722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:48.094680 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d90c80ab-29ef-4083-bd11-74e4ba0c297b","Type":"ContainerStarted","Data":"4dbbf63f431047f6e590358915a5222602183aa24da3a7b4c6d275451ca0180f"} Oct 30 16:21:48.095138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:48.094819 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d90c80ab-29ef-4083-bd11-74e4ba0c297b","Type":"ContainerStarted","Data":"d91b1c2c89a6060cf2a20660dd2ea0761cde8150fe1b8befc226d93f98b1b8b2"} Oct 30 16:21:48.964285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:48.964251 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e33d75-acf8-44b3-a6e3-7599b04fa064" path="/var/lib/kubelet/pods/a6e33d75-acf8-44b3-a6e3-7599b04fa064/volumes" Oct 30 16:21:49.099806 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:49.099771 6400 generic.go:356] "Generic (PLEG): container finished" podID="d90c80ab-29ef-4083-bd11-74e4ba0c297b" containerID="4dbbf63f431047f6e590358915a5222602183aa24da3a7b4c6d275451ca0180f" exitCode=0 Oct 30 16:21:49.100168 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:49.099818 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d90c80ab-29ef-4083-bd11-74e4ba0c297b","Type":"ContainerDied","Data":"4dbbf63f431047f6e590358915a5222602183aa24da3a7b4c6d275451ca0180f"} Oct 30 16:21:50.106681 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:50.106648 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d90c80ab-29ef-4083-bd11-74e4ba0c297b","Type":"ContainerStarted","Data":"a14c1d9d35615bcd90f602a75c4eaf6659ac5eb5b3754dfad6348d1dd79611e5"} Oct 30 16:21:50.107140 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:50.106713 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d90c80ab-29ef-4083-bd11-74e4ba0c297b","Type":"ContainerStarted","Data":"d491adb9495fab8ab68109f17f6939aa4014478bf5e8af8375e29beae7c01d5a"} Oct 30 16:21:50.107140 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:50.106739 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d90c80ab-29ef-4083-bd11-74e4ba0c297b","Type":"ContainerStarted","Data":"1907d099ec3bf4c9ec4796ca4dc7111094a4eb41854dd5db5e3d07f7c5a15376"} Oct 30 16:21:50.107140 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:50.106766 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d90c80ab-29ef-4083-bd11-74e4ba0c297b","Type":"ContainerStarted","Data":"53d4be440634a3e0456622fa9e591b49e7e9d4e44541dd38a84faed4f4ccdb11"} Oct 30 16:21:50.107140 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:50.106796 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d90c80ab-29ef-4083-bd11-74e4ba0c297b","Type":"ContainerStarted","Data":"2cc7ad4aa520779152d10c5d15b50b3751dc6d4514a5cd2386741ddff3b12501"} Oct 30 16:21:50.107140 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:50.106823 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"d90c80ab-29ef-4083-bd11-74e4ba0c297b","Type":"ContainerStarted","Data":"17de3aaf373af851b213c83ef4b0832d92c8132f68e7dc99e83ea7be5349a6ab"} Oct 30 16:21:50.165636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:50.165571 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.161003164 podStartE2EDuration="3.165556947s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.553783ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:21:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:21:50.164719093 +0000 UTC m=+19.315000082" watchObservedRunningTime="2026-10-30 16:21:50.165556947 +0000 UTC m=+19.315837956" Oct 30 16:21:51.441730 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:51.441701 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:21:51.736837 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:51.736782 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:51.737000 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:51.736882 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:51.741533 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:51.741505 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:52.118712 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:52.118626 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-57cc7b5857-4p6bm" Oct 30 16:21:52.192689 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:52.192654 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-565777cb7f-d862k"] Oct 30 16:21:52.653083 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:52.653047 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-glrfx" Oct 30 16:21:54.596196 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:54.596161 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:21:56.992720 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:56.992665 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-d8cfd65cf-hkkx4" podUID="b8014993-7131-499c-98b3-57b1b7429729" containerName="console" containerID="cri-o://011ef189e36172aeb318c7ab823e71b80b8dce90ce6d3b544ffb956605000904" gracePeriod=15 Oct 30 16:21:57.135048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.135020 6400 generic.go:356] "Generic (PLEG): container finished" podID="b8014993-7131-499c-98b3-57b1b7429729" containerID="011ef189e36172aeb318c7ab823e71b80b8dce90ce6d3b544ffb956605000904" exitCode=0 Oct 30 16:21:57.135203 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.135097 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8cfd65cf-hkkx4" event={"ID":"b8014993-7131-499c-98b3-57b1b7429729","Type":"ContainerDied","Data":"011ef189e36172aeb318c7ab823e71b80b8dce90ce6d3b544ffb956605000904"} Oct 30 16:21:57.231587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.231566 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:57.304309 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304226 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-oauth-config\") pod \"b8014993-7131-499c-98b3-57b1b7429729\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " Oct 30 16:21:57.304309 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304276 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-console-config\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-console-config\") pod \"b8014993-7131-499c-98b3-57b1b7429729\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " Oct 30 16:21:57.304537 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304328 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-trusted-ca-bundle\") pod \"b8014993-7131-499c-98b3-57b1b7429729\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " Oct 30 16:21:57.304537 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304356 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8014993-7131-499c-98b3-57b1b7429729-kube-api-access-5kmlq\" (UniqueName: \"kubernetes.io/projected/b8014993-7131-499c-98b3-57b1b7429729-kube-api-access-5kmlq\") pod \"b8014993-7131-499c-98b3-57b1b7429729\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " Oct 30 16:21:57.304537 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304499 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8014993-7131-499c-98b3-57b1b7429729-tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8014993-7131-499c-98b3-57b1b7429729-tmp\") pod \"b8014993-7131-499c-98b3-57b1b7429729\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " Oct 30 16:21:57.304729 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304573 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-serving-cert\") pod \"b8014993-7131-499c-98b3-57b1b7429729\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " Oct 30 16:21:57.304729 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304689 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-service-ca\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-service-ca\") pod \"b8014993-7131-499c-98b3-57b1b7429729\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " Oct 30 16:21:57.304854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304734 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-oauth-serving-cert\") pod \"b8014993-7131-499c-98b3-57b1b7429729\" (UID: \"b8014993-7131-499c-98b3-57b1b7429729\") " Oct 30 16:21:57.304854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304795 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8014993-7131-499c-98b3-57b1b7429729-tmp" pod "b8014993-7131-499c-98b3-57b1b7429729" (UID: "b8014993-7131-499c-98b3-57b1b7429729"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:21:57.304854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304818 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-console-config" pod "b8014993-7131-499c-98b3-57b1b7429729" (UID: "b8014993-7131-499c-98b3-57b1b7429729"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:57.304854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.304799 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-trusted-ca-bundle" pod "b8014993-7131-499c-98b3-57b1b7429729" (UID: "b8014993-7131-499c-98b3-57b1b7429729"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:57.305143 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.305079 6400 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-console-config\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:57.305143 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.305101 6400 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-trusted-ca-bundle\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:57.305143 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.305126 6400 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b8014993-7131-499c-98b3-57b1b7429729-tmp\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:57.305143 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.305101 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-oauth-serving-cert" pod "b8014993-7131-499c-98b3-57b1b7429729" (UID: "b8014993-7131-499c-98b3-57b1b7429729"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:57.305143 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.305184 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-service-ca" pod "b8014993-7131-499c-98b3-57b1b7429729" (UID: "b8014993-7131-499c-98b3-57b1b7429729"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:57.306842 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.306814 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-oauth-config" pod "b8014993-7131-499c-98b3-57b1b7429729" (UID: "b8014993-7131-499c-98b3-57b1b7429729"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:57.306963 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.306858 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8014993-7131-499c-98b3-57b1b7429729-kube-api-access-5kmlq" pod "b8014993-7131-499c-98b3-57b1b7429729" (UID: "b8014993-7131-499c-98b3-57b1b7429729"). InnerVolumeSpecName "kube-api-access-5kmlq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:57.307043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.307023 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-serving-cert" pod "b8014993-7131-499c-98b3-57b1b7429729" (UID: "b8014993-7131-499c-98b3-57b1b7429729"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:57.406386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.406351 6400 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-serving-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:57.406386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.406380 6400 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:57.406386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.406391 6400 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/b8014993-7131-499c-98b3-57b1b7429729-oauth-serving-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:57.406638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.406399 6400 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/b8014993-7131-499c-98b3-57b1b7429729-console-oauth-config\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:57.406638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:57.406408 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kmlq\" (UniqueName: \"kubernetes.io/projected/b8014993-7131-499c-98b3-57b1b7429729-kube-api-access-5kmlq\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:58.047324 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.047283 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-c966567fd-rv9sr" podUID="e22030b7-c861-47ee-b7d7-75f6d659e9c2" containerName="console" containerID="cri-o://669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed" gracePeriod=15 Oct 30 16:21:58.143483 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.143452 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-d8cfd65cf-hkkx4" Oct 30 16:21:58.143483 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.143466 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-d8cfd65cf-hkkx4" event={"ID":"b8014993-7131-499c-98b3-57b1b7429729","Type":"ContainerDied","Data":"a74ad2aa4355d3eeb56a53a285b0325202d10f7296bbf4436e85a22a4a76007e"} Oct 30 16:21:58.143738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.143547 6400 scope.go:122] "RemoveContainer" containerID="011ef189e36172aeb318c7ab823e71b80b8dce90ce6d3b544ffb956605000904" Oct 30 16:21:58.187663 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.187633 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-d8cfd65cf-hkkx4"] Oct 30 16:21:58.197123 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.197097 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-d8cfd65cf-hkkx4"] Oct 30 16:21:58.286066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.286044 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:58.412692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.412584 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e22030b7-c861-47ee-b7d7-75f6d659e9c2-kube-api-access-j2dv7\" (UniqueName: \"kubernetes.io/projected/e22030b7-c861-47ee-b7d7-75f6d659e9c2-kube-api-access-j2dv7\") pod \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " Oct 30 16:21:58.412692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.412683 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-serving-cert\") pod \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " Oct 30 16:21:58.412879 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.412725 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-service-ca\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-service-ca\") pod \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " Oct 30 16:21:58.412879 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.412800 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-config\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-config\") pod \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " Oct 30 16:21:58.412879 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.412838 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e22030b7-c861-47ee-b7d7-75f6d659e9c2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22030b7-c861-47ee-b7d7-75f6d659e9c2-tmp\") pod \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " Oct 30 16:21:58.412879 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.412865 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-oauth-config\") pod \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " Oct 30 16:21:58.413070 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.412918 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-oauth-serving-cert\") pod \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\" (UID: \"e22030b7-c861-47ee-b7d7-75f6d659e9c2\") " Oct 30 16:21:58.413177 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.413151 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e22030b7-c861-47ee-b7d7-75f6d659e9c2-tmp" pod "e22030b7-c861-47ee-b7d7-75f6d659e9c2" (UID: "e22030b7-c861-47ee-b7d7-75f6d659e9c2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:21:58.413231 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.413165 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-service-ca" pod "e22030b7-c861-47ee-b7d7-75f6d659e9c2" (UID: "e22030b7-c861-47ee-b7d7-75f6d659e9c2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:58.413310 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.413285 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-config" pod "e22030b7-c861-47ee-b7d7-75f6d659e9c2" (UID: "e22030b7-c861-47ee-b7d7-75f6d659e9c2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:58.413675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.413648 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-oauth-serving-cert" pod "e22030b7-c861-47ee-b7d7-75f6d659e9c2" (UID: "e22030b7-c861-47ee-b7d7-75f6d659e9c2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:21:58.415186 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.415161 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e22030b7-c861-47ee-b7d7-75f6d659e9c2-kube-api-access-j2dv7" pod "e22030b7-c861-47ee-b7d7-75f6d659e9c2" (UID: "e22030b7-c861-47ee-b7d7-75f6d659e9c2"). InnerVolumeSpecName "kube-api-access-j2dv7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:21:58.415186 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.415176 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-serving-cert" pod "e22030b7-c861-47ee-b7d7-75f6d659e9c2" (UID: "e22030b7-c861-47ee-b7d7-75f6d659e9c2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:58.415311 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.415201 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-oauth-config" pod "e22030b7-c861-47ee-b7d7-75f6d659e9c2" (UID: "e22030b7-c861-47ee-b7d7-75f6d659e9c2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:21:58.513857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.513822 6400 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-config\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:58.513857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.513850 6400 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e22030b7-c861-47ee-b7d7-75f6d659e9c2-tmp\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:58.513857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.513859 6400 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-oauth-config\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:58.513857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.513869 6400 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-oauth-serving-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:58.514120 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.513877 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2dv7\" (UniqueName: \"kubernetes.io/projected/e22030b7-c861-47ee-b7d7-75f6d659e9c2-kube-api-access-j2dv7\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:58.514120 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.513888 6400 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/e22030b7-c861-47ee-b7d7-75f6d659e9c2-console-serving-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:58.514120 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.513896 6400 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/e22030b7-c861-47ee-b7d7-75f6d659e9c2-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:21:58.967422 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:58.967388 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8014993-7131-499c-98b3-57b1b7429729" path="/var/lib/kubelet/pods/b8014993-7131-499c-98b3-57b1b7429729/volumes" Oct 30 16:21:59.148905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:59.148869 6400 generic.go:356] "Generic (PLEG): container finished" podID="e22030b7-c861-47ee-b7d7-75f6d659e9c2" containerID="669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed" exitCode=0 Oct 30 16:21:59.149371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:59.148911 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-c966567fd-rv9sr" Oct 30 16:21:59.149371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:59.148926 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c966567fd-rv9sr" event={"ID":"e22030b7-c861-47ee-b7d7-75f6d659e9c2","Type":"ContainerDied","Data":"669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed"} Oct 30 16:21:59.149371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:59.148970 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-c966567fd-rv9sr" event={"ID":"e22030b7-c861-47ee-b7d7-75f6d659e9c2","Type":"ContainerDied","Data":"06f0c2cfa5ed9fff43f86ce77efaaab23ac240158b23d4a3f41d9713ea796c3c"} Oct 30 16:21:59.149371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:59.148989 6400 scope.go:122] "RemoveContainer" containerID="669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed" Oct 30 16:21:59.157058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:59.157038 6400 scope.go:122] "RemoveContainer" containerID="669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed" Oct 30 16:21:59.157330 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:21:59.157310 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed\": container with ID starting with 669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed not found: ID does not exist" containerID="669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed" Oct 30 16:21:59.157379 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:59.157340 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed"} err="failed to get container status \"669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed\": rpc error: code = NotFound desc = could not find container \"669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed\": container with ID starting with 669856cd925e238979eccfbad1b37cc32139ee13ae565c47e067c535532545ed not found: ID does not exist" Oct 30 16:21:59.191210 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:59.191187 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-c966567fd-rv9sr"] Oct 30 16:21:59.200924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:21:59.200904 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-c966567fd-rv9sr"] Oct 30 16:22:00.964171 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:00.964136 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e22030b7-c861-47ee-b7d7-75f6d659e9c2" path="/var/lib/kubelet/pods/e22030b7-c861-47ee-b7d7-75f6d659e9c2/volumes" Oct 30 16:22:05.326481 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:05.326452 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-gccnz" Oct 30 16:22:17.213695 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.213639 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-565777cb7f-d862k" podUID="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" containerName="console" containerID="cri-o://0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4" gracePeriod=15 Oct 30 16:22:17.466788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.466731 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:22:17.577134 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577106 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-service-ca\") pod \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " Oct 30 16:22:17.577305 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577148 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-kube-api-access-z6gbn\" (UniqueName: \"kubernetes.io/projected/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-kube-api-access-z6gbn\") pod \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " Oct 30 16:22:17.577305 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577168 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-config\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-config\") pod \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " Oct 30 16:22:17.577305 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577195 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-oauth-config\") pod \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " Oct 30 16:22:17.577448 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577340 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-tmp\" (UniqueName: \"kubernetes.io/empty-dir/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-tmp\") pod \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " Oct 30 16:22:17.577448 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577386 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-trusted-ca-bundle\") pod \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " Oct 30 16:22:17.577551 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577456 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-oauth-serving-cert\") pod \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " Oct 30 16:22:17.577551 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577499 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-serving-cert\") pod \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\" (UID: \"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8\") " Oct 30 16:22:17.577693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577667 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-config" pod "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" (UID: "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:22:17.577753 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577694 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-service-ca" pod "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" (UID: "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:22:17.577877 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577853 6400 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-config\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:22:17.577877 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577865 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-tmp" pod "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" (UID: "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:22:17.578283 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577878 6400 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:22:17.578283 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577913 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-trusted-ca-bundle" pod "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" (UID: "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:22:17.578283 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.577953 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-oauth-serving-cert" pod "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" (UID: "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:22:17.579774 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.579754 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-kube-api-access-z6gbn" pod "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" (UID: "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8"). InnerVolumeSpecName "kube-api-access-z6gbn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:22:17.579883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.579767 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-oauth-config" pod "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" (UID: "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:22:17.579883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.579842 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-serving-cert" pod "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" (UID: "a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:22:17.678942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.678895 6400 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-serving-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:22:17.678942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.678936 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6gbn\" (UniqueName: \"kubernetes.io/projected/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-kube-api-access-z6gbn\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:22:17.678942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.678949 6400 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-console-oauth-config\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:22:17.678942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.678958 6400 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-tmp\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:22:17.679210 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.678969 6400 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-trusted-ca-bundle\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:22:17.679210 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:17.678978 6400 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8-oauth-serving-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:22:18.221231 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:18.221196 6400 generic.go:356] "Generic (PLEG): container finished" podID="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" containerID="0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4" exitCode=0 Oct 30 16:22:18.221660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:18.221249 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-565777cb7f-d862k" Oct 30 16:22:18.221660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:18.221262 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565777cb7f-d862k" event={"ID":"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8","Type":"ContainerDied","Data":"0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4"} Oct 30 16:22:18.221660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:18.221332 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-565777cb7f-d862k" event={"ID":"a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8","Type":"ContainerDied","Data":"32da6cefad5f9764cf3178c7c6878e4ded6d15d81574464735d04eb0da6a5703"} Oct 30 16:22:18.221660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:18.221353 6400 scope.go:122] "RemoveContainer" containerID="0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4" Oct 30 16:22:18.230122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:18.230107 6400 scope.go:122] "RemoveContainer" containerID="0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4" Oct 30 16:22:18.230363 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:22:18.230344 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4\": container with ID starting with 0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4 not found: ID does not exist" containerID="0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4" Oct 30 16:22:18.230416 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:18.230391 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4"} err="failed to get container status \"0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4\": rpc error: code = NotFound desc = could not find container \"0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4\": container with ID starting with 0b229f0811f05d980b0c518bcabe42d4f39cf0c601c1e8d684adf1d2da58d3e4 not found: ID does not exist" Oct 30 16:22:18.263708 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:18.263673 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-565777cb7f-d862k"] Oct 30 16:22:18.273386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:18.273352 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-565777cb7f-d862k"] Oct 30 16:22:18.965101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:22:18.965067 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" path="/var/lib/kubelet/pods/a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8/volumes" Oct 30 16:22:24.596577 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:22:24.596522 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:22:54.595408 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:22:54.595372 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:23:24.596577 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:23:24.596535 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:23:54.596106 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:23:54.596066 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:24:13.636583 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.636489 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c"] Oct 30 16:24:13.637148 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.637137 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e22030b7-c861-47ee-b7d7-75f6d659e9c2" containerName="console" Oct 30 16:24:13.637211 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.637155 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e22030b7-c861-47ee-b7d7-75f6d659e9c2" containerName="console" Oct 30 16:24:13.637211 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.637194 6400 cpu_manager.go:414] "containerMap: removing container" podUID="b8014993-7131-499c-98b3-57b1b7429729" containerName="console" Oct 30 16:24:13.637211 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.637201 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8014993-7131-499c-98b3-57b1b7429729" containerName="console" Oct 30 16:24:13.637348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.637236 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" containerName="console" Oct 30 16:24:13.637348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.637244 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" containerName="console" Oct 30 16:24:13.637441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.637360 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8014993-7131-499c-98b3-57b1b7429729" containerName="console" Oct 30 16:24:13.637441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.637375 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53ecbbc-c6c6-40ed-9054-94e5d25a8ea8" containerName="console" Oct 30 16:24:13.637441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.637391 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e22030b7-c861-47ee-b7d7-75f6d659e9c2" containerName="console" Oct 30 16:24:13.641485 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.641462 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:13.663788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.663765 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:24:13.663788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.663770 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:24:13.663788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.663770 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-d64t6\"" Oct 30 16:24:13.674855 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.674832 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c"] Oct 30 16:24:13.747402 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.747368 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:13.747402 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.747406 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:13.747642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.747449 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnhgf\" (UniqueName: \"kubernetes.io/projected/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-kube-api-access-lnhgf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:13.848755 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.848695 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:13.848936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.848769 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:13.848936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.848842 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnhgf\" (UniqueName: \"kubernetes.io/projected/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-kube-api-access-lnhgf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:13.849105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.849081 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:13.849162 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.849106 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:13.870192 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.870166 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnhgf\" (UniqueName: \"kubernetes.io/projected/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-kube-api-access-lnhgf\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:13.954167 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:13.954089 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:14.246730 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:14.246698 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c"] Oct 30 16:24:14.250251 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:24:14.250218 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29ec56f_771f_4e7f_92bb_1b7fadc2ed6d.slice/crio-c89aaeea621b0bbaf10dfd35f8fb2c57e580775c9fab3fc4bc6d4008b5bfcf66 WatchSource:0}: Error finding container c89aaeea621b0bbaf10dfd35f8fb2c57e580775c9fab3fc4bc6d4008b5bfcf66: Status 404 returned error can't find the container with id c89aaeea621b0bbaf10dfd35f8fb2c57e580775c9fab3fc4bc6d4008b5bfcf66 Oct 30 16:24:14.252356 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:14.252340 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:24:14.635910 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:14.635824 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" event={"ID":"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d","Type":"ContainerStarted","Data":"c89aaeea621b0bbaf10dfd35f8fb2c57e580775c9fab3fc4bc6d4008b5bfcf66"} Oct 30 16:24:19.656335 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:19.656296 6400 generic.go:356] "Generic (PLEG): container finished" podID="a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" containerID="34739c5053118bbdd8156159a88eb86a875dcd7ab45a6a53d1386d1871600d23" exitCode=0 Oct 30 16:24:19.656814 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:19.656353 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" event={"ID":"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d","Type":"ContainerDied","Data":"34739c5053118bbdd8156159a88eb86a875dcd7ab45a6a53d1386d1871600d23"} Oct 30 16:24:21.666670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:21.666590 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" event={"ID":"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d","Type":"ContainerStarted","Data":"61fdd4d46a06c5da570aea99731de7d2695f3827e2c22b755afa2198e88b7058"} Oct 30 16:24:22.671363 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:22.671325 6400 generic.go:356] "Generic (PLEG): container finished" podID="a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" containerID="61fdd4d46a06c5da570aea99731de7d2695f3827e2c22b755afa2198e88b7058" exitCode=0 Oct 30 16:24:22.671762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:22.671408 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" event={"ID":"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d","Type":"ContainerDied","Data":"61fdd4d46a06c5da570aea99731de7d2695f3827e2c22b755afa2198e88b7058"} Oct 30 16:24:24.603915 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:24:24.603876 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:24:28.696137 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:28.696109 6400 generic.go:356] "Generic (PLEG): container finished" podID="a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" containerID="058b42302c11cce544d78b76363a04eb083a73de2348c57b00ade35103a4170a" exitCode=0 Oct 30 16:24:28.696443 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:28.696195 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" event={"ID":"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d","Type":"ContainerDied","Data":"058b42302c11cce544d78b76363a04eb083a73de2348c57b00ade35103a4170a"} Oct 30 16:24:29.823960 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:29.823934 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:29.883826 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:29.883793 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-kube-api-access-lnhgf\" (UniqueName: \"kubernetes.io/projected/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-kube-api-access-lnhgf\") pod \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " Oct 30 16:24:29.883982 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:29.883884 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-bundle\" (UniqueName: \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-bundle\") pod \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " Oct 30 16:24:29.883982 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:29.883921 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-util\" (UniqueName: \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-util\") pod \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\" (UID: \"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d\") " Oct 30 16:24:29.884523 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:29.884500 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-bundle" pod "a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" (UID: "a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:24:29.886159 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:29.886136 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-kube-api-access-lnhgf" pod "a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" (UID: "a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d"). InnerVolumeSpecName "kube-api-access-lnhgf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:24:29.888348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:29.888320 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-util" pod "a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" (UID: "a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:24:29.984577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:29.984548 6400 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-bundle\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:24:29.984577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:29.984578 6400 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-util\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:24:29.984792 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:29.984587 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnhgf\" (UniqueName: \"kubernetes.io/projected/a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d-kube-api-access-lnhgf\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:24:30.704885 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:30.704853 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" event={"ID":"a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d","Type":"ContainerDied","Data":"c89aaeea621b0bbaf10dfd35f8fb2c57e580775c9fab3fc4bc6d4008b5bfcf66"} Oct 30 16:24:30.704885 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:30.704875 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651c2878c" Oct 30 16:24:30.705084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:30.704897 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c89aaeea621b0bbaf10dfd35f8fb2c57e580775c9fab3fc4bc6d4008b5bfcf66" Oct 30 16:24:36.777405 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.777372 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs"] Oct 30 16:24:36.777818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.777698 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" containerName="util" Oct 30 16:24:36.777818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.777711 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" containerName="util" Oct 30 16:24:36.777818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.777729 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" containerName="pull" Oct 30 16:24:36.777818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.777734 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" containerName="pull" Oct 30 16:24:36.777818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.777754 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" containerName="extract" Oct 30 16:24:36.777818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.777760 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" containerName="extract" Oct 30 16:24:36.777818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.777815 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29ec56f-771f-4e7f-92bb-1b7fadc2ed6d" containerName="extract" Oct 30 16:24:36.811978 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.811949 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs"] Oct 30 16:24:36.812152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.812043 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" Oct 30 16:24:36.834580 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.834555 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-xq5dm\"" Oct 30 16:24:36.837672 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.837640 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:24:36.844232 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.844219 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\"" Oct 30 16:24:36.942428 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.942394 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/076a16e5-eb8f-4580-bb22-d6d6b081c792-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jn9qs\" (UID: \"076a16e5-eb8f-4580-bb22-d6d6b081c792\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" Oct 30 16:24:36.942639 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:36.942439 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pz9w\" (UniqueName: \"kubernetes.io/projected/076a16e5-eb8f-4580-bb22-d6d6b081c792-kube-api-access-7pz9w\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jn9qs\" (UID: \"076a16e5-eb8f-4580-bb22-d6d6b081c792\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" Oct 30 16:24:37.043784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:37.043706 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pz9w\" (UniqueName: \"kubernetes.io/projected/076a16e5-eb8f-4580-bb22-d6d6b081c792-kube-api-access-7pz9w\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jn9qs\" (UID: \"076a16e5-eb8f-4580-bb22-d6d6b081c792\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" Oct 30 16:24:37.043784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:37.043778 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/076a16e5-eb8f-4580-bb22-d6d6b081c792-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jn9qs\" (UID: \"076a16e5-eb8f-4580-bb22-d6d6b081c792\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" Oct 30 16:24:37.044095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:37.044077 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/076a16e5-eb8f-4580-bb22-d6d6b081c792-tmp\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jn9qs\" (UID: \"076a16e5-eb8f-4580-bb22-d6d6b081c792\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" Oct 30 16:24:37.069959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:37.069932 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pz9w\" (UniqueName: \"kubernetes.io/projected/076a16e5-eb8f-4580-bb22-d6d6b081c792-kube-api-access-7pz9w\") pod \"cert-manager-operator-controller-manager-89fdbf7dc-jn9qs\" (UID: \"076a16e5-eb8f-4580-bb22-d6d6b081c792\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" Oct 30 16:24:37.136091 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:37.136060 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" Oct 30 16:24:37.478674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:37.478581 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs"] Oct 30 16:24:37.482251 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:24:37.482226 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod076a16e5_eb8f_4580_bb22_d6d6b081c792.slice/crio-b262797e948533b3c0d6d14eae5c504c38aece72a2b061fad2b6ca5fadb86122 WatchSource:0}: Error finding container b262797e948533b3c0d6d14eae5c504c38aece72a2b061fad2b6ca5fadb86122: Status 404 returned error can't find the container with id b262797e948533b3c0d6d14eae5c504c38aece72a2b061fad2b6ca5fadb86122 Oct 30 16:24:37.731114 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:37.731027 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" event={"ID":"076a16e5-eb8f-4580-bb22-d6d6b081c792","Type":"ContainerStarted","Data":"b262797e948533b3c0d6d14eae5c504c38aece72a2b061fad2b6ca5fadb86122"} Oct 30 16:24:40.745495 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:40.745423 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" event={"ID":"076a16e5-eb8f-4580-bb22-d6d6b081c792","Type":"ContainerStarted","Data":"be397055afc9b1f26afb37670933bdd6aea5f27ece915ba7aec490d0b006ef35"} Oct 30 16:24:40.809781 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:40.809726 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-89fdbf7dc-jn9qs" podStartSLOduration=2.5285454659999997 podStartE2EDuration="4.809711514s" totalImagesPullingTime="2.281166048s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:40.799783738 +0000 UTC m=+189.950064726" watchObservedRunningTime="2026-10-30 16:24:40.809711514 +0000 UTC m=+189.959992499" Oct 30 16:24:41.235395 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.235365 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl"] Oct 30 16:24:41.253453 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.253427 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" Oct 30 16:24:41.262083 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.262057 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl"] Oct 30 16:24:41.269822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.269796 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 16:24:41.269951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.269861 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 16:24:41.269951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.269894 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-nx98m\"" Oct 30 16:24:41.381346 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.381317 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13d78a22-e592-4fa7-b371-4261c6e5c152-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-w66gl\" (UID: \"13d78a22-e592-4fa7-b371-4261c6e5c152\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" Oct 30 16:24:41.381531 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.381353 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qvpb8\" (UniqueName: \"kubernetes.io/projected/13d78a22-e592-4fa7-b371-4261c6e5c152-kube-api-access-qvpb8\") pod \"cert-manager-webhook-5dbbbf75cc-w66gl\" (UID: \"13d78a22-e592-4fa7-b371-4261c6e5c152\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" Oct 30 16:24:41.482292 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.482253 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13d78a22-e592-4fa7-b371-4261c6e5c152-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-w66gl\" (UID: \"13d78a22-e592-4fa7-b371-4261c6e5c152\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" Oct 30 16:24:41.482292 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.482292 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qvpb8\" (UniqueName: \"kubernetes.io/projected/13d78a22-e592-4fa7-b371-4261c6e5c152-kube-api-access-qvpb8\") pod \"cert-manager-webhook-5dbbbf75cc-w66gl\" (UID: \"13d78a22-e592-4fa7-b371-4261c6e5c152\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" Oct 30 16:24:41.512582 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.512509 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qvpb8\" (UniqueName: \"kubernetes.io/projected/13d78a22-e592-4fa7-b371-4261c6e5c152-kube-api-access-qvpb8\") pod \"cert-manager-webhook-5dbbbf75cc-w66gl\" (UID: \"13d78a22-e592-4fa7-b371-4261c6e5c152\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" Oct 30 16:24:41.530905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.530873 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/13d78a22-e592-4fa7-b371-4261c6e5c152-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-w66gl\" (UID: \"13d78a22-e592-4fa7-b371-4261c6e5c152\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" Oct 30 16:24:41.589547 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.589520 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" Oct 30 16:24:41.738710 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.738681 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx"] Oct 30 16:24:41.764182 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.764103 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:41.768380 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.768281 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx"] Oct 30 16:24:41.785570 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.785549 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:24:41.785570 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.785564 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:24:41.785744 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.785667 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"61b1ff68c7255ee5c5329fe19a5d96293a9578-ce91a85d-dockercfg-8xtsc\"" Oct 30 16:24:41.884530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.884501 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:41.884712 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.884560 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:41.884841 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.884819 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pstfm\" (UniqueName: \"kubernetes.io/projected/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-kube-api-access-pstfm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:41.901471 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.901355 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl"] Oct 30 16:24:41.904430 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:24:41.904389 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13d78a22_e592_4fa7_b371_4261c6e5c152.slice/crio-bd96074788b15be29c0d303c3e137cd31660f2e326ae3cea5a710efb55ad6300 WatchSource:0}: Error finding container bd96074788b15be29c0d303c3e137cd31660f2e326ae3cea5a710efb55ad6300: Status 404 returned error can't find the container with id bd96074788b15be29c0d303c3e137cd31660f2e326ae3cea5a710efb55ad6300 Oct 30 16:24:41.986166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.986127 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pstfm\" (UniqueName: \"kubernetes.io/projected/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-kube-api-access-pstfm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:41.986358 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.986184 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:41.986358 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.986219 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:41.986548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.986531 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-util\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:41.986633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:41.986593 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-bundle\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:42.007451 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:42.007416 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pstfm\" (UniqueName: \"kubernetes.io/projected/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-kube-api-access-pstfm\") pod \"61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:42.117401 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:42.117331 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:42.439405 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:42.439113 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx"] Oct 30 16:24:42.441845 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:24:42.441817 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod617b92d8_39ee_4c22_9ff1_c8a3c5fd0b41.slice/crio-ef1bac1ecad8f035575f139fa3c3c28b1d5ea944dd1c4b9bf669bbb6881a1dcb WatchSource:0}: Error finding container ef1bac1ecad8f035575f139fa3c3c28b1d5ea944dd1c4b9bf669bbb6881a1dcb: Status 404 returned error can't find the container with id ef1bac1ecad8f035575f139fa3c3c28b1d5ea944dd1c4b9bf669bbb6881a1dcb Oct 30 16:24:42.756169 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:42.756121 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" event={"ID":"13d78a22-e592-4fa7-b371-4261c6e5c152","Type":"ContainerStarted","Data":"bd96074788b15be29c0d303c3e137cd31660f2e326ae3cea5a710efb55ad6300"} Oct 30 16:24:42.757943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:42.757913 6400 generic.go:356] "Generic (PLEG): container finished" podID="617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" containerID="d7e40e68f7879e419a171097fac9fae074f8b2ccfb75f293065e35122e3f5835" exitCode=0 Oct 30 16:24:42.758069 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:42.758003 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" event={"ID":"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41","Type":"ContainerDied","Data":"d7e40e68f7879e419a171097fac9fae074f8b2ccfb75f293065e35122e3f5835"} Oct 30 16:24:42.758069 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:42.758046 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" event={"ID":"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41","Type":"ContainerStarted","Data":"ef1bac1ecad8f035575f139fa3c3c28b1d5ea944dd1c4b9bf669bbb6881a1dcb"} Oct 30 16:24:45.772315 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:45.772277 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" event={"ID":"13d78a22-e592-4fa7-b371-4261c6e5c152","Type":"ContainerStarted","Data":"2be8fc647a7de564ce012bb18b691e20a5503d4e9358b36cbd6f303f6b0ff989"} Oct 30 16:24:45.772788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:45.772335 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" Oct 30 16:24:45.774082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:45.774055 6400 generic.go:356] "Generic (PLEG): container finished" podID="617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" containerID="9a0fc42a86aa3e244301b950c7838b6739fa5d9f747fafbe855459a4e4297007" exitCode=0 Oct 30 16:24:45.774160 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:45.774141 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" event={"ID":"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41","Type":"ContainerDied","Data":"9a0fc42a86aa3e244301b950c7838b6739fa5d9f747fafbe855459a4e4297007"} Oct 30 16:24:45.809337 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:45.809287 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" podStartSLOduration=1.866031257 podStartE2EDuration="4.809269286s" totalImagesPullingTime="2.943238029s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:45.808679032 +0000 UTC m=+194.958960020" watchObservedRunningTime="2026-10-30 16:24:45.809269286 +0000 UTC m=+194.959550274" Oct 30 16:24:46.780660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:46.780628 6400 generic.go:356] "Generic (PLEG): container finished" podID="617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" containerID="91b6e93da7652a5474a7fe004021dc0a75e928ee50d67affdd69f0e954df7852" exitCode=0 Oct 30 16:24:46.781094 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:46.780697 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" event={"ID":"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41","Type":"ContainerDied","Data":"91b6e93da7652a5474a7fe004021dc0a75e928ee50d67affdd69f0e954df7852"} Oct 30 16:24:47.914373 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:47.914350 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:48.039060 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.038985 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-kube-api-access-pstfm\" (UniqueName: \"kubernetes.io/projected/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-kube-api-access-pstfm\") pod \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " Oct 30 16:24:48.039060 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.039044 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-bundle\" (UniqueName: \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-bundle\") pod \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " Oct 30 16:24:48.039299 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.039065 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-util\" (UniqueName: \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-util\") pod \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\" (UID: \"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41\") " Oct 30 16:24:48.039493 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.039470 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-bundle" pod "617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" (UID: "617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:24:48.041264 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.041236 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-kube-api-access-pstfm" pod "617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" (UID: "617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41"). InnerVolumeSpecName "kube-api-access-pstfm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:24:48.044005 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.043963 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-util" pod "617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" (UID: "617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:24:48.140320 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.140289 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pstfm\" (UniqueName: \"kubernetes.io/projected/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-kube-api-access-pstfm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:24:48.140320 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.140314 6400 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-bundle\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:24:48.140320 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.140322 6400 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41-util\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:24:48.790200 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.790168 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" event={"ID":"617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41","Type":"ContainerDied","Data":"ef1bac1ecad8f035575f139fa3c3c28b1d5ea944dd1c4b9bf669bbb6881a1dcb"} Oct 30 16:24:48.790200 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.790187 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/61b1ff68c7255ee5c5329fe19a5d96293a95788d9734a0a79829e8dc855d9vx" Oct 30 16:24:48.790415 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:48.790212 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ef1bac1ecad8f035575f139fa3c3c28b1d5ea944dd1c4b9bf669bbb6881a1dcb" Oct 30 16:24:51.649469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.649436 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-p994p"] Oct 30 16:24:51.650038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.649794 6400 cpu_manager.go:414] "containerMap: removing container" podUID="617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" containerName="util" Oct 30 16:24:51.650038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.649812 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" containerName="util" Oct 30 16:24:51.650038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.649835 6400 cpu_manager.go:414] "containerMap: removing container" podUID="617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" containerName="extract" Oct 30 16:24:51.650038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.649842 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" containerName="extract" Oct 30 16:24:51.650038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.649867 6400 cpu_manager.go:414] "containerMap: removing container" podUID="617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" containerName="pull" Oct 30 16:24:51.650038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.649874 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" containerName="pull" Oct 30 16:24:51.650038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.649951 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="617b92d8-39ee-4c22-9ff1-c8a3c5fd0b41" containerName="extract" Oct 30 16:24:51.652073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.652056 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-p994p" Oct 30 16:24:51.673795 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.673775 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-txjxn\"" Oct 30 16:24:51.676325 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.676308 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-p994p"] Oct 30 16:24:51.773304 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.773267 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5b7c7ad-6f42-4878-bc65-622d4412b2ed-bound-sa-token\") pod \"cert-manager-597b95cb84-p994p\" (UID: \"e5b7c7ad-6f42-4878-bc65-622d4412b2ed\") " pod="cert-manager/cert-manager-597b95cb84-p994p" Oct 30 16:24:51.773473 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.773314 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2wg5\" (UniqueName: \"kubernetes.io/projected/e5b7c7ad-6f42-4878-bc65-622d4412b2ed-kube-api-access-q2wg5\") pod \"cert-manager-597b95cb84-p994p\" (UID: \"e5b7c7ad-6f42-4878-bc65-622d4412b2ed\") " pod="cert-manager/cert-manager-597b95cb84-p994p" Oct 30 16:24:51.783089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.783063 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-w66gl" Oct 30 16:24:51.874730 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.874696 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5b7c7ad-6f42-4878-bc65-622d4412b2ed-bound-sa-token\") pod \"cert-manager-597b95cb84-p994p\" (UID: \"e5b7c7ad-6f42-4878-bc65-622d4412b2ed\") " pod="cert-manager/cert-manager-597b95cb84-p994p" Oct 30 16:24:51.874914 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.874753 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2wg5\" (UniqueName: \"kubernetes.io/projected/e5b7c7ad-6f42-4878-bc65-622d4412b2ed-kube-api-access-q2wg5\") pod \"cert-manager-597b95cb84-p994p\" (UID: \"e5b7c7ad-6f42-4878-bc65-622d4412b2ed\") " pod="cert-manager/cert-manager-597b95cb84-p994p" Oct 30 16:24:51.897213 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.897186 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e5b7c7ad-6f42-4878-bc65-622d4412b2ed-bound-sa-token\") pod \"cert-manager-597b95cb84-p994p\" (UID: \"e5b7c7ad-6f42-4878-bc65-622d4412b2ed\") " pod="cert-manager/cert-manager-597b95cb84-p994p" Oct 30 16:24:51.897351 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.897237 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2wg5\" (UniqueName: \"kubernetes.io/projected/e5b7c7ad-6f42-4878-bc65-622d4412b2ed-kube-api-access-q2wg5\") pod \"cert-manager-597b95cb84-p994p\" (UID: \"e5b7c7ad-6f42-4878-bc65-622d4412b2ed\") " pod="cert-manager/cert-manager-597b95cb84-p994p" Oct 30 16:24:51.963107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:51.963025 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-p994p" Oct 30 16:24:52.254862 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:52.254827 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-p994p"] Oct 30 16:24:52.258172 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:24:52.258145 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode5b7c7ad_6f42_4878_bc65_622d4412b2ed.slice/crio-292e2c91f3c1ae9e1f0c630e0159382c524f966a4af3b898c1e8f442d2dbdec2 WatchSource:0}: Error finding container 292e2c91f3c1ae9e1f0c630e0159382c524f966a4af3b898c1e8f442d2dbdec2: Status 404 returned error can't find the container with id 292e2c91f3c1ae9e1f0c630e0159382c524f966a4af3b898c1e8f442d2dbdec2 Oct 30 16:24:52.806945 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:52.806902 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-p994p" event={"ID":"e5b7c7ad-6f42-4878-bc65-622d4412b2ed","Type":"ContainerStarted","Data":"de1d22d869cd5e6197b29e90e10619625df3cd8b3fb5d18c05ff3686b2e53436"} Oct 30 16:24:52.807339 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:52.806953 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-p994p" event={"ID":"e5b7c7ad-6f42-4878-bc65-622d4412b2ed","Type":"ContainerStarted","Data":"292e2c91f3c1ae9e1f0c630e0159382c524f966a4af3b898c1e8f442d2dbdec2"} Oct 30 16:24:52.842903 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:52.842852 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-p994p" podStartSLOduration=1.8428375689999998 podStartE2EDuration="1.842837569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:52.842036469 +0000 UTC m=+201.992317458" watchObservedRunningTime="2026-10-30 16:24:52.842837569 +0000 UTC m=+201.993118556" Oct 30 16:24:54.596747 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:24:54.596708 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:24:54.613535 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.613453 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-trjhp"] Oct 30 16:24:54.616075 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.616054 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" Oct 30 16:24:54.641056 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.641014 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-p5zwx\"" Oct 30 16:24:54.641056 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.641032 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\"" Oct 30 16:24:54.641056 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.641048 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\"" Oct 30 16:24:54.642186 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.642167 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-trjhp"] Oct 30 16:24:54.702322 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.702284 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc7fr\" (UniqueName: \"kubernetes.io/projected/cd5838d2-29ad-43df-9b8c-1a7326b28d44-kube-api-access-kc7fr\") pod \"openshift-lws-operator-8465994787-trjhp\" (UID: \"cd5838d2-29ad-43df-9b8c-1a7326b28d44\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" Oct 30 16:24:54.702521 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.702405 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd5838d2-29ad-43df-9b8c-1a7326b28d44-tmp\") pod \"openshift-lws-operator-8465994787-trjhp\" (UID: \"cd5838d2-29ad-43df-9b8c-1a7326b28d44\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" Oct 30 16:24:54.803095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.803056 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd5838d2-29ad-43df-9b8c-1a7326b28d44-tmp\") pod \"openshift-lws-operator-8465994787-trjhp\" (UID: \"cd5838d2-29ad-43df-9b8c-1a7326b28d44\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" Oct 30 16:24:54.803275 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.803122 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc7fr\" (UniqueName: \"kubernetes.io/projected/cd5838d2-29ad-43df-9b8c-1a7326b28d44-kube-api-access-kc7fr\") pod \"openshift-lws-operator-8465994787-trjhp\" (UID: \"cd5838d2-29ad-43df-9b8c-1a7326b28d44\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" Oct 30 16:24:54.803463 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.803444 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd5838d2-29ad-43df-9b8c-1a7326b28d44-tmp\") pod \"openshift-lws-operator-8465994787-trjhp\" (UID: \"cd5838d2-29ad-43df-9b8c-1a7326b28d44\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" Oct 30 16:24:54.837546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.837516 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc7fr\" (UniqueName: \"kubernetes.io/projected/cd5838d2-29ad-43df-9b8c-1a7326b28d44-kube-api-access-kc7fr\") pod \"openshift-lws-operator-8465994787-trjhp\" (UID: \"cd5838d2-29ad-43df-9b8c-1a7326b28d44\") " pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" Oct 30 16:24:54.926465 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:54.926390 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" Oct 30 16:24:55.220244 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:55.220160 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-8465994787-trjhp"] Oct 30 16:24:55.223276 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:24:55.223244 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcd5838d2_29ad_43df_9b8c_1a7326b28d44.slice/crio-50b266768d21ef24c1013d4e4cbe02decc3c98a195b83479e147c3040f462062 WatchSource:0}: Error finding container 50b266768d21ef24c1013d4e4cbe02decc3c98a195b83479e147c3040f462062: Status 404 returned error can't find the container with id 50b266768d21ef24c1013d4e4cbe02decc3c98a195b83479e147c3040f462062 Oct 30 16:24:55.821316 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:55.821280 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" event={"ID":"cd5838d2-29ad-43df-9b8c-1a7326b28d44","Type":"ContainerStarted","Data":"50b266768d21ef24c1013d4e4cbe02decc3c98a195b83479e147c3040f462062"} Oct 30 16:24:57.831416 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:57.831379 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" event={"ID":"cd5838d2-29ad-43df-9b8c-1a7326b28d44","Type":"ContainerStarted","Data":"66d66a8a3136c2a5240683203dbbb1aad69ac794575ff90a10b2d11653ce802e"} Oct 30 16:24:57.868278 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:24:57.868228 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-8465994787-trjhp" podStartSLOduration=2.270649926 podStartE2EDuration="3.868214621s" totalImagesPullingTime="1.597564695s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:24:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:24:57.867649273 +0000 UTC m=+207.017930260" watchObservedRunningTime="2026-10-30 16:24:57.868214621 +0000 UTC m=+207.018495608" Oct 30 16:25:11.957106 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:11.957038 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt"] Oct 30 16:25:11.959726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:11.959704 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:11.981965 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:11.981941 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:25:11.982101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:11.982078 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:25:11.982204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:11.982186 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-jt94s\"" Oct 30 16:25:11.983384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:11.983367 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt"] Oct 30 16:25:12.050994 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.050957 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:12.051207 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.051102 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:12.051207 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.051144 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dmzb\" (UniqueName: \"kubernetes.io/projected/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-kube-api-access-6dmzb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:12.151925 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.151888 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:12.152117 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.151943 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:12.152117 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.151968 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dmzb\" (UniqueName: \"kubernetes.io/projected/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-kube-api-access-6dmzb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:12.152342 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.152317 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:12.152411 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.152329 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:12.173786 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.173754 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dmzb\" (UniqueName: \"kubernetes.io/projected/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-kube-api-access-6dmzb\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:12.273029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.272993 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:12.567490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.567384 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt"] Oct 30 16:25:12.570785 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:25:12.570754 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99e6e5b_1f6a_4561_8a8d_15f7d9448941.slice/crio-bc9976580a5b3ce81111f5dc6d54be909c2009cc06161afc4a33fc497060edc3 WatchSource:0}: Error finding container bc9976580a5b3ce81111f5dc6d54be909c2009cc06161afc4a33fc497060edc3: Status 404 returned error can't find the container with id bc9976580a5b3ce81111f5dc6d54be909c2009cc06161afc4a33fc497060edc3 Oct 30 16:25:12.893198 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.893101 6400 generic.go:356] "Generic (PLEG): container finished" podID="e99e6e5b-1f6a-4561-8a8d-15f7d9448941" containerID="5f6ed4f50b8104a2b0a2c53e66c7bc751cf7a57d345a0728ef7d8a4fdba3e42a" exitCode=0 Oct 30 16:25:12.893198 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.893168 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" event={"ID":"e99e6e5b-1f6a-4561-8a8d-15f7d9448941","Type":"ContainerDied","Data":"5f6ed4f50b8104a2b0a2c53e66c7bc751cf7a57d345a0728ef7d8a4fdba3e42a"} Oct 30 16:25:12.893382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:12.893209 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" event={"ID":"e99e6e5b-1f6a-4561-8a8d-15f7d9448941","Type":"ContainerStarted","Data":"bc9976580a5b3ce81111f5dc6d54be909c2009cc06161afc4a33fc497060edc3"} Oct 30 16:25:13.899521 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:13.899488 6400 generic.go:356] "Generic (PLEG): container finished" podID="e99e6e5b-1f6a-4561-8a8d-15f7d9448941" containerID="9d7ec2e1d6d1a5014accff8fbbc9365e1249776113d16f3629c42e8795f9e772" exitCode=0 Oct 30 16:25:13.899941 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:13.899575 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" event={"ID":"e99e6e5b-1f6a-4561-8a8d-15f7d9448941","Type":"ContainerDied","Data":"9d7ec2e1d6d1a5014accff8fbbc9365e1249776113d16f3629c42e8795f9e772"} Oct 30 16:25:14.906995 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:14.906951 6400 generic.go:356] "Generic (PLEG): container finished" podID="e99e6e5b-1f6a-4561-8a8d-15f7d9448941" containerID="cb354ee5e63171ec5aa2cc0aa42f7e94365e3ddb56350b1cb2877a7fe463cdad" exitCode=0 Oct 30 16:25:14.907387 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:14.907043 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" event={"ID":"e99e6e5b-1f6a-4561-8a8d-15f7d9448941","Type":"ContainerDied","Data":"cb354ee5e63171ec5aa2cc0aa42f7e94365e3ddb56350b1cb2877a7fe463cdad"} Oct 30 16:25:15.082063 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.082032 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7"] Oct 30 16:25:15.084558 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.084532 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.110735 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.110703 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7"] Oct 30 16:25:15.165275 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.165179 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\"" Oct 30 16:25:15.165275 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.165248 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\"" Oct 30 16:25:15.165275 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.165276 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-fmsm8\"" Oct 30 16:25:15.169302 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.169288 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\"" Oct 30 16:25:15.180731 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.180706 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-cert\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.180833 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.180769 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-manager-config\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.180899 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.180856 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-metrics-cert\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.180959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.180944 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jghl7\" (UniqueName: \"kubernetes.io/projected/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-kube-api-access-jghl7\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.281860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.281821 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-manager-config\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.281860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.281858 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-metrics-cert\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.282073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.281990 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jghl7\" (UniqueName: \"kubernetes.io/projected/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-kube-api-access-jghl7\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.282073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.282059 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-cert\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.282692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.282673 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-manager-config\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.284480 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.284458 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-metrics-cert\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.284565 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.284495 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-cert\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.304117 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.304093 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jghl7\" (UniqueName: \"kubernetes.io/projected/eb84c8dc-4cd6-4a38-9035-eebb83c176a0-kube-api-access-jghl7\") pod \"lws-controller-manager-6584f6464-nkjn7\" (UID: \"eb84c8dc-4cd6-4a38-9035-eebb83c176a0\") " pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.394949 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.394904 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:15.689976 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.689942 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7"] Oct 30 16:25:15.693484 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:25:15.693397 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb84c8dc_4cd6_4a38_9035_eebb83c176a0.slice/crio-6eec0f058daa41bf87bd002264acdadc39040412e5a2af93c569b9b791332ce2 WatchSource:0}: Error finding container 6eec0f058daa41bf87bd002264acdadc39040412e5a2af93c569b9b791332ce2: Status 404 returned error can't find the container with id 6eec0f058daa41bf87bd002264acdadc39040412e5a2af93c569b9b791332ce2 Oct 30 16:25:15.912233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:15.912194 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" event={"ID":"eb84c8dc-4cd6-4a38-9035-eebb83c176a0","Type":"ContainerStarted","Data":"6eec0f058daa41bf87bd002264acdadc39040412e5a2af93c569b9b791332ce2"} Oct 30 16:25:16.038619 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.038578 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:16.190200 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.190166 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-bundle\" (UniqueName: \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-bundle\") pod \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " Oct 30 16:25:16.190389 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.190220 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-util\" (UniqueName: \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-util\") pod \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " Oct 30 16:25:16.190389 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.190270 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-kube-api-access-6dmzb\" (UniqueName: \"kubernetes.io/projected/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-kube-api-access-6dmzb\") pod \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\" (UID: \"e99e6e5b-1f6a-4561-8a8d-15f7d9448941\") " Oct 30 16:25:16.190891 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.190863 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-bundle" pod "e99e6e5b-1f6a-4561-8a8d-15f7d9448941" (UID: "e99e6e5b-1f6a-4561-8a8d-15f7d9448941"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:25:16.192538 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.192512 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-kube-api-access-6dmzb" pod "e99e6e5b-1f6a-4561-8a8d-15f7d9448941" (UID: "e99e6e5b-1f6a-4561-8a8d-15f7d9448941"). InnerVolumeSpecName "kube-api-access-6dmzb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:25:16.195693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.195661 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-util" pod "e99e6e5b-1f6a-4561-8a8d-15f7d9448941" (UID: "e99e6e5b-1f6a-4561-8a8d-15f7d9448941"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:25:16.291806 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.291777 6400 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-bundle\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:25:16.291917 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.291812 6400 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-util\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:25:16.291917 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.291825 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dmzb\" (UniqueName: \"kubernetes.io/projected/e99e6e5b-1f6a-4561-8a8d-15f7d9448941-kube-api-access-6dmzb\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:25:16.928965 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.928923 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" event={"ID":"e99e6e5b-1f6a-4561-8a8d-15f7d9448941","Type":"ContainerDied","Data":"bc9976580a5b3ce81111f5dc6d54be909c2009cc06161afc4a33fc497060edc3"} Oct 30 16:25:16.929404 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.928973 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71kqjvt" Oct 30 16:25:16.929404 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:16.929002 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc9976580a5b3ce81111f5dc6d54be909c2009cc06161afc4a33fc497060edc3" Oct 30 16:25:17.934465 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:17.934425 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" event={"ID":"eb84c8dc-4cd6-4a38-9035-eebb83c176a0","Type":"ContainerStarted","Data":"e01bde4cff1263ea34a665195ea68e32eed6b1cfa76383ed3cff4f35ed4b57bc"} Oct 30 16:25:17.934893 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:17.934565 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:17.971465 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:17.971407 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" podStartSLOduration=1.582170478 podStartE2EDuration="2.971388198s" totalImagesPullingTime="1.38921772s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:17.970329951 +0000 UTC m=+227.120610938" watchObservedRunningTime="2026-10-30 16:25:17.971388198 +0000 UTC m=+227.121669187" Oct 30 16:25:24.598093 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:25:24.598052 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:25:24.989886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:24.989852 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn"] Oct 30 16:25:24.990190 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:24.990177 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e99e6e5b-1f6a-4561-8a8d-15f7d9448941" containerName="util" Oct 30 16:25:24.990239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:24.990192 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99e6e5b-1f6a-4561-8a8d-15f7d9448941" containerName="util" Oct 30 16:25:24.990239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:24.990209 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e99e6e5b-1f6a-4561-8a8d-15f7d9448941" containerName="pull" Oct 30 16:25:24.990239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:24.990215 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99e6e5b-1f6a-4561-8a8d-15f7d9448941" containerName="pull" Oct 30 16:25:24.990239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:24.990222 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e99e6e5b-1f6a-4561-8a8d-15f7d9448941" containerName="extract" Oct 30 16:25:24.990239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:24.990226 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e99e6e5b-1f6a-4561-8a8d-15f7d9448941" containerName="extract" Oct 30 16:25:24.990386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:24.990283 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e99e6e5b-1f6a-4561-8a8d-15f7d9448941" containerName="extract" Oct 30 16:25:24.992411 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:24.992394 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.032326 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.032286 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn"] Oct 30 16:25:25.069029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.068998 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 16:25:25.069029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.069018 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 16:25:25.069029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.069000 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-metrics-tls\"" Oct 30 16:25:25.069029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.069014 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wc8qt\"" Oct 30 16:25:25.069822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.069801 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 16:25:25.070076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.070059 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 16:25:25.166894 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.166860 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-tmp\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.166894 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.166899 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-manifests\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.167155 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.166987 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-webhook-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.167155 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.167065 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-metrics-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.167155 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.167107 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5dql\" (UniqueName: \"kubernetes.io/projected/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-kube-api-access-w5dql\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.167155 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.167144 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.267635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.267515 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.267635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.267580 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-tmp\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.267872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.267639 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-manifests\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.267872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.267695 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-webhook-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.267872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.267722 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-metrics-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.267872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.267761 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5dql\" (UniqueName: \"kubernetes.io/projected/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-kube-api-access-w5dql\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.268173 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.268142 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-manifests\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.270353 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.270331 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-webhook-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.270449 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.270360 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-metrics-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.270449 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.270371 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-apiservice-cert\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.270519 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.270454 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-tmp\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.293944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.293922 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5dql\" (UniqueName: \"kubernetes.io/projected/5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22-kube-api-access-w5dql\") pod \"opendatahub-operator-controller-manager-cbcd98d6d-xpdtn\" (UID: \"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22\") " pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.304741 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.304719 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:25.655475 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.655340 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn"] Oct 30 16:25:25.657881 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:25:25.657857 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5269c1d7_fe1e_40c5_9a17_4f6d2fbe2f22.slice/crio-52e8cf7ea93652c9c13b2de6b8f6b4c10802583d41833ff747c8418dd654a2ec WatchSource:0}: Error finding container 52e8cf7ea93652c9c13b2de6b8f6b4c10802583d41833ff747c8418dd654a2ec: Status 404 returned error can't find the container with id 52e8cf7ea93652c9c13b2de6b8f6b4c10802583d41833ff747c8418dd654a2ec Oct 30 16:25:25.972867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:25.972781 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" event={"ID":"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22","Type":"ContainerStarted","Data":"52e8cf7ea93652c9c13b2de6b8f6b4c10802583d41833ff747c8418dd654a2ec"} Oct 30 16:25:28.941521 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:28.941491 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-6584f6464-nkjn7" Oct 30 16:25:28.989266 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:28.989234 6400 generic.go:356] "Generic (PLEG): container finished" podID="5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22" containerID="411fc00272e7faee60a162cb155a3f60d71a0b55f11d02ffcde155a3832614bb" exitCode=0 Oct 30 16:25:28.989435 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:28.989327 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" event={"ID":"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22","Type":"ContainerDied","Data":"411fc00272e7faee60a162cb155a3f60d71a0b55f11d02ffcde155a3832614bb"} Oct 30 16:25:29.996450 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:29.996415 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" event={"ID":"5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22","Type":"ContainerStarted","Data":"bc06449093c4ffd1771d77f96e18b9acf4b55c87c6e721f4669761cabd524f81"} Oct 30 16:25:29.996969 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:29.996522 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:30.078148 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:30.078097 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" podStartSLOduration=2.511523158 podStartE2EDuration="6.078083367s" totalImagesPullingTime="3.148057657s" totalInitContainerRuntime="418.502552ms" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:30.067145337 +0000 UTC m=+239.217426325" watchObservedRunningTime="2026-10-30 16:25:30.078083367 +0000 UTC m=+239.228364349" Oct 30 16:25:41.002691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:41.002659 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-cbcd98d6d-xpdtn" Oct 30 16:25:48.311551 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.311514 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-observability-584f99c788-k7579"] Oct 30 16:25:48.315494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.315471 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:25:48.333184 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.333161 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-webhook-cert\"" Oct 30 16:25:48.342999 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.342978 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-observability-dockercfg-wbqb7\"" Oct 30 16:25:48.344856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.344821 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-k7579"] Oct 30 16:25:48.345529 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.345492 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/02309873-b9e3-4457-85d2-75c94131b212-webhook-certs\") pod \"odh-observability-584f99c788-k7579\" (UID: \"02309873-b9e3-4457-85d2-75c94131b212\") " pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:25:48.345667 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.345544 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wptn\" (UniqueName: \"kubernetes.io/projected/02309873-b9e3-4457-85d2-75c94131b212-kube-api-access-7wptn\") pod \"odh-observability-584f99c788-k7579\" (UID: \"02309873-b9e3-4457-85d2-75c94131b212\") " pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:25:48.447017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.446982 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/02309873-b9e3-4457-85d2-75c94131b212-webhook-certs\") pod \"odh-observability-584f99c788-k7579\" (UID: \"02309873-b9e3-4457-85d2-75c94131b212\") " pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:25:48.447207 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.447029 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wptn\" (UniqueName: \"kubernetes.io/projected/02309873-b9e3-4457-85d2-75c94131b212-kube-api-access-7wptn\") pod \"odh-observability-584f99c788-k7579\" (UID: \"02309873-b9e3-4457-85d2-75c94131b212\") " pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:25:48.447207 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:25:48.447142 6400 secret.go:189] Couldn't get secret opendatahub/odh-observability-webhook-cert: secret "odh-observability-webhook-cert" not found Oct 30 16:25:48.447299 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:25:48.447216 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/02309873-b9e3-4457-85d2-75c94131b212-webhook-certs podName:02309873-b9e3-4457-85d2-75c94131b212 nodeName:}" failed. No retries permitted until 2026-10-30 16:25:48.947200947 +0000 UTC m=+258.097481931 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-certs" (UniqueName: "kubernetes.io/secret/02309873-b9e3-4457-85d2-75c94131b212-webhook-certs") pod "odh-observability-584f99c788-k7579" (UID: "02309873-b9e3-4457-85d2-75c94131b212") : secret "odh-observability-webhook-cert" not found Oct 30 16:25:48.470991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.470957 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wptn\" (UniqueName: \"kubernetes.io/projected/02309873-b9e3-4457-85d2-75c94131b212-kube-api-access-7wptn\") pod \"odh-observability-584f99c788-k7579\" (UID: \"02309873-b9e3-4457-85d2-75c94131b212\") " pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:25:48.951829 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.951798 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/02309873-b9e3-4457-85d2-75c94131b212-webhook-certs\") pod \"odh-observability-584f99c788-k7579\" (UID: \"02309873-b9e3-4457-85d2-75c94131b212\") " pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:25:48.954635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:48.954586 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-certs\" (UniqueName: \"kubernetes.io/secret/02309873-b9e3-4457-85d2-75c94131b212-webhook-certs\") pod \"odh-observability-584f99c788-k7579\" (UID: \"02309873-b9e3-4457-85d2-75c94131b212\") " pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:25:49.228009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:49.227922 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:25:49.523957 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:49.523424 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-observability-584f99c788-k7579"] Oct 30 16:25:49.525886 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:25:49.525851 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02309873_b9e3_4457_85d2_75c94131b212.slice/crio-ffa45b4e8c528f541d4b42e839ac92d4cf3035f412beab29e6c0e35633a838d8 WatchSource:0}: Error finding container ffa45b4e8c528f541d4b42e839ac92d4cf3035f412beab29e6c0e35633a838d8: Status 404 returned error can't find the container with id ffa45b4e8c528f541d4b42e839ac92d4cf3035f412beab29e6c0e35633a838d8 Oct 30 16:25:50.086646 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:50.086587 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-k7579" event={"ID":"02309873-b9e3-4457-85d2-75c94131b212","Type":"ContainerStarted","Data":"ffa45b4e8c528f541d4b42e839ac92d4cf3035f412beab29e6c0e35633a838d8"} Oct 30 16:25:51.554977 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.554940 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9"] Oct 30 16:25:51.558125 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.558099 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.571444 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.571421 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 16:25:51.574033 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.574005 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.574299 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.574273 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.574385 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.574328 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.574385 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.574357 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hr7t7\" (UniqueName: \"kubernetes.io/projected/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-kube-api-access-hr7t7\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.574494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.574393 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.574494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.574449 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.574494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.574475 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.574752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.574505 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.574752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.574548 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.576772 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.576748 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 16:25:51.588737 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.588710 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9"] Oct 30 16:25:51.623462 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.623412 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-qjrkg\"" Oct 30 16:25:51.675490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.675454 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.675698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.675500 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hr7t7\" (UniqueName: \"kubernetes.io/projected/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-kube-api-access-hr7t7\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.675698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.675526 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.675698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.675558 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.675698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.675587 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.675698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.675623 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.675698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.675658 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.675698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.675678 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.676078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.675755 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.676078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.676003 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.676078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.676035 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-data\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.676231 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.676188 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.676292 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.676242 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.677885 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.677863 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 16:25:51.678434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.678413 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.678942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.678922 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.687210 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.687182 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.697168 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.697145 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-istio-token\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.697285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.697189 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hr7t7\" (UniqueName: \"kubernetes.io/projected/ffda5415-4ff3-4fce-abfc-c8e44f905c5b-kube-api-access-hr7t7\") pod \"data-science-gateway-data-science-gateway-class-67c76445c-kz9k9\" (UID: \"ffda5415-4ff3-4fce-abfc-c8e44f905c5b\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:51.880782 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:51.880748 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:52.097820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:52.097728 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-observability-584f99c788-k7579" event={"ID":"02309873-b9e3-4457-85d2-75c94131b212","Type":"ContainerStarted","Data":"c0d95b90b5847272e40284e12ad39faabec2ec0bd9b175631a361cfd41edc713"} Oct 30 16:25:52.097997 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:52.097879 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:25:52.134624 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:52.134541 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-observability-584f99c788-k7579" podStartSLOduration=1.88079962 podStartE2EDuration="4.134520843s" totalImagesPullingTime="2.253721223s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:52.134242218 +0000 UTC m=+261.284523207" watchObservedRunningTime="2026-10-30 16:25:52.134520843 +0000 UTC m=+261.284801872" Oct 30 16:25:52.189143 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:52.189116 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9"] Oct 30 16:25:52.192129 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:25:52.192103 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffda5415_4ff3_4fce_abfc_c8e44f905c5b.slice/crio-db3f0ebc42bad9ad8b5a4d59c7b5c68ecab4b062a53d05ecedbfa81ecae4a902 WatchSource:0}: Error finding container db3f0ebc42bad9ad8b5a4d59c7b5c68ecab4b062a53d05ecedbfa81ecae4a902: Status 404 returned error can't find the container with id db3f0ebc42bad9ad8b5a4d59c7b5c68ecab4b062a53d05ecedbfa81ecae4a902 Oct 30 16:25:53.105089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:53.105046 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" event={"ID":"ffda5415-4ff3-4fce-abfc-c8e44f905c5b","Type":"ContainerStarted","Data":"db3f0ebc42bad9ad8b5a4d59c7b5c68ecab4b062a53d05ecedbfa81ecae4a902"} Oct 30 16:25:54.600351 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:25:54.600292 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:25:54.909227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:54.909195 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:25:54.909329 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:54.909270 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:25:54.909329 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:54.909297 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:25:55.116061 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:55.116026 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" event={"ID":"ffda5415-4ff3-4fce-abfc-c8e44f905c5b","Type":"ContainerStarted","Data":"de3d75890c3f7148cfe6b95c49e4aa68a7ddfebac71c423edca0e90dcf9e0190"} Oct 30 16:25:55.176215 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:55.176121 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" podStartSLOduration=1.462089583 podStartE2EDuration="4.176104975s" totalImagesPullingTime="2.714015392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:25:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:25:55.165483061 +0000 UTC m=+264.315764080" watchObservedRunningTime="2026-10-30 16:25:55.176104975 +0000 UTC m=+264.326385962" Oct 30 16:25:55.881425 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:55.881391 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:55.886770 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:55.886744 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:56.120464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:56.120427 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:25:56.121517 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:25:56.121499 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-67c76445c-kz9k9" Oct 30 16:26:03.107592 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:03.107560 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-observability-584f99c788-k7579" Oct 30 16:26:24.597325 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:26:24.597289 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:26:47.921143 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:47.921102 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vg6k7"] Oct 30 16:26:47.924094 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:47.924074 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" Oct 30 16:26:47.943370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:47.943346 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 16:26:47.943702 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:47.943686 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-4z6kq\"" Oct 30 16:26:47.954003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:47.953973 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vg6k7"] Oct 30 16:26:47.972320 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:47.972283 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv"] Oct 30 16:26:47.975302 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:47.975284 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:26:47.997257 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:47.997224 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv"] Oct 30 16:26:48.005392 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.005364 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d953cc88-99a1-4389-aec1-f8cfda8e7de5-cert\") pod \"kserve-controller-manager-776cc66d47-vg6k7\" (UID: \"d953cc88-99a1-4389-aec1-f8cfda8e7de5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" Oct 30 16:26:48.005523 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.005399 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtt5n\" (UniqueName: \"kubernetes.io/projected/d953cc88-99a1-4389-aec1-f8cfda8e7de5-kube-api-access-wtt5n\") pod \"kserve-controller-manager-776cc66d47-vg6k7\" (UID: \"d953cc88-99a1-4389-aec1-f8cfda8e7de5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" Oct 30 16:26:48.026468 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.026436 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-zgr4b"] Oct 30 16:26:48.028926 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.028907 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" Oct 30 16:26:48.045806 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.045780 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\"" Oct 30 16:26:48.045806 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.045780 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-j6nhp\"" Oct 30 16:26:48.045996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.045856 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 16:26:48.045996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.045916 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-kkbkl\"" Oct 30 16:26:48.050266 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.050244 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-zgr4b"] Oct 30 16:26:48.055787 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.055766 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-5fmhl"] Oct 30 16:26:48.058580 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.058560 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.080984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.080960 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-5fmhl"] Oct 30 16:26:48.106344 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.106312 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbfdk\" (UniqueName: \"kubernetes.io/projected/ed0131d6-c205-4bd0-9169-63656361235d-kube-api-access-pbfdk\") pod \"llmisvc-controller-manager-869d5dcc5d-nxcfv\" (UID: \"ed0131d6-c205-4bd0-9169-63656361235d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:26:48.106490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.106354 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d953cc88-99a1-4389-aec1-f8cfda8e7de5-cert\") pod \"kserve-controller-manager-776cc66d47-vg6k7\" (UID: \"d953cc88-99a1-4389-aec1-f8cfda8e7de5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" Oct 30 16:26:48.106490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.106374 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtt5n\" (UniqueName: \"kubernetes.io/projected/d953cc88-99a1-4389-aec1-f8cfda8e7de5-kube-api-access-wtt5n\") pod \"kserve-controller-manager-776cc66d47-vg6k7\" (UID: \"d953cc88-99a1-4389-aec1-f8cfda8e7de5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" Oct 30 16:26:48.106490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.106450 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa757e1d-7dbd-47cf-bc6f-54dacda93285-tls-certs\") pod \"model-serving-api-75bd89fdfc-zgr4b\" (UID: \"aa757e1d-7dbd-47cf-bc6f-54dacda93285\") " pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" Oct 30 16:26:48.106699 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.106500 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2f4p\" (UniqueName: \"kubernetes.io/projected/aa757e1d-7dbd-47cf-bc6f-54dacda93285-kube-api-access-z2f4p\") pod \"model-serving-api-75bd89fdfc-zgr4b\" (UID: \"aa757e1d-7dbd-47cf-bc6f-54dacda93285\") " pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" Oct 30 16:26:48.106699 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.106587 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed0131d6-c205-4bd0-9169-63656361235d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nxcfv\" (UID: \"ed0131d6-c205-4bd0-9169-63656361235d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:26:48.109061 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.109038 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d953cc88-99a1-4389-aec1-f8cfda8e7de5-cert\") pod \"kserve-controller-manager-776cc66d47-vg6k7\" (UID: \"d953cc88-99a1-4389-aec1-f8cfda8e7de5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" Oct 30 16:26:48.128302 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.128279 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtt5n\" (UniqueName: \"kubernetes.io/projected/d953cc88-99a1-4389-aec1-f8cfda8e7de5-kube-api-access-wtt5n\") pod \"kserve-controller-manager-776cc66d47-vg6k7\" (UID: \"d953cc88-99a1-4389-aec1-f8cfda8e7de5\") " pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" Oct 30 16:26:48.144437 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.144413 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 16:26:48.144833 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.144816 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 16:26:48.144923 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.144909 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-chpc9\"" Oct 30 16:26:48.207516 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.207435 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ac575cb-7c53-489b-b05c-09ca444bfabb-cert\") pod \"odh-model-controller-f87fcf56c-5fmhl\" (UID: \"7ac575cb-7c53-489b-b05c-09ca444bfabb\") " pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.207516 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.207495 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa757e1d-7dbd-47cf-bc6f-54dacda93285-tls-certs\") pod \"model-serving-api-75bd89fdfc-zgr4b\" (UID: \"aa757e1d-7dbd-47cf-bc6f-54dacda93285\") " pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" Oct 30 16:26:48.207772 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.207572 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ac575cb-7c53-489b-b05c-09ca444bfabb-metrics-certs\") pod \"odh-model-controller-f87fcf56c-5fmhl\" (UID: \"7ac575cb-7c53-489b-b05c-09ca444bfabb\") " pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.207772 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.207697 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2f4p\" (UniqueName: \"kubernetes.io/projected/aa757e1d-7dbd-47cf-bc6f-54dacda93285-kube-api-access-z2f4p\") pod \"model-serving-api-75bd89fdfc-zgr4b\" (UID: \"aa757e1d-7dbd-47cf-bc6f-54dacda93285\") " pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" Oct 30 16:26:48.207772 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.207756 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zl4qp\" (UniqueName: \"kubernetes.io/projected/7ac575cb-7c53-489b-b05c-09ca444bfabb-kube-api-access-zl4qp\") pod \"odh-model-controller-f87fcf56c-5fmhl\" (UID: \"7ac575cb-7c53-489b-b05c-09ca444bfabb\") " pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.207933 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.207795 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed0131d6-c205-4bd0-9169-63656361235d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nxcfv\" (UID: \"ed0131d6-c205-4bd0-9169-63656361235d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:26:48.207933 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.207867 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbfdk\" (UniqueName: \"kubernetes.io/projected/ed0131d6-c205-4bd0-9169-63656361235d-kube-api-access-pbfdk\") pod \"llmisvc-controller-manager-869d5dcc5d-nxcfv\" (UID: \"ed0131d6-c205-4bd0-9169-63656361235d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:26:48.210260 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.210232 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa757e1d-7dbd-47cf-bc6f-54dacda93285-tls-certs\") pod \"model-serving-api-75bd89fdfc-zgr4b\" (UID: \"aa757e1d-7dbd-47cf-bc6f-54dacda93285\") " pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" Oct 30 16:26:48.210397 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.210298 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed0131d6-c205-4bd0-9169-63656361235d-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-nxcfv\" (UID: \"ed0131d6-c205-4bd0-9169-63656361235d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:26:48.229168 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.229142 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbfdk\" (UniqueName: \"kubernetes.io/projected/ed0131d6-c205-4bd0-9169-63656361235d-kube-api-access-pbfdk\") pod \"llmisvc-controller-manager-869d5dcc5d-nxcfv\" (UID: \"ed0131d6-c205-4bd0-9169-63656361235d\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:26:48.234417 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.234377 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2f4p\" (UniqueName: \"kubernetes.io/projected/aa757e1d-7dbd-47cf-bc6f-54dacda93285-kube-api-access-z2f4p\") pod \"model-serving-api-75bd89fdfc-zgr4b\" (UID: \"aa757e1d-7dbd-47cf-bc6f-54dacda93285\") " pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" Oct 30 16:26:48.237258 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.237237 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" Oct 30 16:26:48.287269 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.286552 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:26:48.308886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.308854 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ac575cb-7c53-489b-b05c-09ca444bfabb-cert\") pod \"odh-model-controller-f87fcf56c-5fmhl\" (UID: \"7ac575cb-7c53-489b-b05c-09ca444bfabb\") " pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.309048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.308913 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ac575cb-7c53-489b-b05c-09ca444bfabb-metrics-certs\") pod \"odh-model-controller-f87fcf56c-5fmhl\" (UID: \"7ac575cb-7c53-489b-b05c-09ca444bfabb\") " pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.309048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.309000 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zl4qp\" (UniqueName: \"kubernetes.io/projected/7ac575cb-7c53-489b-b05c-09ca444bfabb-kube-api-access-zl4qp\") pod \"odh-model-controller-f87fcf56c-5fmhl\" (UID: \"7ac575cb-7c53-489b-b05c-09ca444bfabb\") " pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.311836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.311809 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7ac575cb-7c53-489b-b05c-09ca444bfabb-cert\") pod \"odh-model-controller-f87fcf56c-5fmhl\" (UID: \"7ac575cb-7c53-489b-b05c-09ca444bfabb\") " pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.312145 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.312129 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ac575cb-7c53-489b-b05c-09ca444bfabb-metrics-certs\") pod \"odh-model-controller-f87fcf56c-5fmhl\" (UID: \"7ac575cb-7c53-489b-b05c-09ca444bfabb\") " pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.333038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.333007 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zl4qp\" (UniqueName: \"kubernetes.io/projected/7ac575cb-7c53-489b-b05c-09ca444bfabb-kube-api-access-zl4qp\") pod \"odh-model-controller-f87fcf56c-5fmhl\" (UID: \"7ac575cb-7c53-489b-b05c-09ca444bfabb\") " pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.341297 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.340907 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" Oct 30 16:26:48.372750 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.372725 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:48.537938 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.537802 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-vg6k7"] Oct 30 16:26:48.540542 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:26:48.540515 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd953cc88_99a1_4389_aec1_f8cfda8e7de5.slice/crio-fea0af70eb15df9ae32f88a165a704d7014599df9ebe284ea172ce03b6e0d765 WatchSource:0}: Error finding container fea0af70eb15df9ae32f88a165a704d7014599df9ebe284ea172ce03b6e0d765: Status 404 returned error can't find the container with id fea0af70eb15df9ae32f88a165a704d7014599df9ebe284ea172ce03b6e0d765 Oct 30 16:26:48.589485 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.589363 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv"] Oct 30 16:26:48.591722 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:26:48.591688 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-poded0131d6_c205_4bd0_9169_63656361235d.slice/crio-5861694ec602b0bb6e53b8272523edc69a10e4e3ebc46540f8f02175b2cfa404 WatchSource:0}: Error finding container 5861694ec602b0bb6e53b8272523edc69a10e4e3ebc46540f8f02175b2cfa404: Status 404 returned error can't find the container with id 5861694ec602b0bb6e53b8272523edc69a10e4e3ebc46540f8f02175b2cfa404 Oct 30 16:26:48.645085 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.645055 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-zgr4b"] Oct 30 16:26:48.649426 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:26:48.649377 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa757e1d_7dbd_47cf_bc6f_54dacda93285.slice/crio-1b6723b340ce25c46b558256b385a1c252634a97221c133177f52f7037605279 WatchSource:0}: Error finding container 1b6723b340ce25c46b558256b385a1c252634a97221c133177f52f7037605279: Status 404 returned error can't find the container with id 1b6723b340ce25c46b558256b385a1c252634a97221c133177f52f7037605279 Oct 30 16:26:48.687433 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:48.687357 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-f87fcf56c-5fmhl"] Oct 30 16:26:48.696467 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:26:48.696438 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ac575cb_7c53_489b_b05c_09ca444bfabb.slice/crio-02c53be2ce7bb50e6a269a177e39b46d46609ec749917704c58c5f31c2d5800c WatchSource:0}: Error finding container 02c53be2ce7bb50e6a269a177e39b46d46609ec749917704c58c5f31c2d5800c: Status 404 returned error can't find the container with id 02c53be2ce7bb50e6a269a177e39b46d46609ec749917704c58c5f31c2d5800c Oct 30 16:26:49.401366 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:49.400860 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" event={"ID":"ed0131d6-c205-4bd0-9169-63656361235d","Type":"ContainerStarted","Data":"5861694ec602b0bb6e53b8272523edc69a10e4e3ebc46540f8f02175b2cfa404"} Oct 30 16:26:49.404165 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:49.404099 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" event={"ID":"7ac575cb-7c53-489b-b05c-09ca444bfabb","Type":"ContainerStarted","Data":"02c53be2ce7bb50e6a269a177e39b46d46609ec749917704c58c5f31c2d5800c"} Oct 30 16:26:49.412993 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:49.412959 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" event={"ID":"d953cc88-99a1-4389-aec1-f8cfda8e7de5","Type":"ContainerStarted","Data":"fea0af70eb15df9ae32f88a165a704d7014599df9ebe284ea172ce03b6e0d765"} Oct 30 16:26:49.416431 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:49.416369 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" event={"ID":"aa757e1d-7dbd-47cf-bc6f-54dacda93285","Type":"ContainerStarted","Data":"1b6723b340ce25c46b558256b385a1c252634a97221c133177f52f7037605279"} Oct 30 16:26:52.447447 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:52.447336 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" event={"ID":"aa757e1d-7dbd-47cf-bc6f-54dacda93285","Type":"ContainerStarted","Data":"0a64c84d99b2d2321a645757828ccdc1b319fa8466697ea5274377e2450648f1"} Oct 30 16:26:52.448200 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:52.448137 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" Oct 30 16:26:52.485767 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:52.485702 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" podStartSLOduration=2.449936038 podStartE2EDuration="5.485671277s" totalImagesPullingTime="3.035735239s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:26:52.485391197 +0000 UTC m=+321.635672185" watchObservedRunningTime="2026-10-30 16:26:52.485671277 +0000 UTC m=+321.635952266" Oct 30 16:26:53.453593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:53.453553 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" event={"ID":"ed0131d6-c205-4bd0-9169-63656361235d","Type":"ContainerStarted","Data":"d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6"} Oct 30 16:26:53.454163 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:53.453756 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:26:53.455281 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:53.455256 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" event={"ID":"7ac575cb-7c53-489b-b05c-09ca444bfabb","Type":"ContainerStarted","Data":"3c97c5ea3d692c2fa7b42c48f20b09cabe41eade136098be6044753f1af134d2"} Oct 30 16:26:53.455409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:53.455306 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:26:53.456847 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:53.456825 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" event={"ID":"d953cc88-99a1-4389-aec1-f8cfda8e7de5","Type":"ContainerStarted","Data":"072f0e239562b96538cf09cc16656b7b3a74be875a26ed8168d3566c2488571e"} Oct 30 16:26:53.490519 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:53.490425 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" podStartSLOduration=2.1531665 podStartE2EDuration="6.49041089s" totalImagesPullingTime="4.33724439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:26:53.489253156 +0000 UTC m=+322.639534144" watchObservedRunningTime="2026-10-30 16:26:53.49041089 +0000 UTC m=+322.640691880" Oct 30 16:26:53.525840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:53.525782 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" podStartSLOduration=2.286471394 podStartE2EDuration="6.525764602s" totalImagesPullingTime="4.239293208s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:26:53.524508583 +0000 UTC m=+322.674789572" watchObservedRunningTime="2026-10-30 16:26:53.525764602 +0000 UTC m=+322.676045591" Oct 30 16:26:53.561169 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:53.561115 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" podStartSLOduration=1.381773406 podStartE2EDuration="5.561100232s" totalImagesPullingTime="4.179326826s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:26:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:26:53.559685369 +0000 UTC m=+322.709966360" watchObservedRunningTime="2026-10-30 16:26:53.561100232 +0000 UTC m=+322.711381220" Oct 30 16:26:54.462146 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:26:54.462115 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" Oct 30 16:26:54.598312 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:26:54.598271 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:27:04.464644 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:04.464552 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-f87fcf56c-5fmhl" Oct 30 16:27:04.466437 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:04.466416 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-zgr4b" Oct 30 16:27:24.464546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:24.464515 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:27:24.597709 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:27:24.597672 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:27:25.471288 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:25.471258 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-vg6k7" Oct 30 16:27:30.979801 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:30.979774 6400 scope.go:122] "RemoveContainer" containerID="dcceb272812eb79c1bbd2ab10113ad8ea84053a65ab531aefe91c7b2a8977add" Oct 30 16:27:30.987510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:30.987494 6400 scope.go:122] "RemoveContainer" containerID="16fdba6dda4782e29fc5b9866a5d9e996ff8737fb6b47d979212a7bb40e79415" Oct 30 16:27:30.994666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:30.994643 6400 scope.go:122] "RemoveContainer" containerID="3b12bba989d729d6102b23b4ab112389882af4ea2af10d31b16ac16790b89ed9" Oct 30 16:27:31.001682 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:31.001663 6400 scope.go:122] "RemoveContainer" containerID="52a15ac6f58e4511905538277767d83c9d89e40a552b1dcdcad33de6f837429f" Oct 30 16:27:31.009304 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:31.009286 6400 scope.go:122] "RemoveContainer" containerID="a2e5530d27d52d0b4ff9b07d782b7281dc740e638b0f0d7d2e3c973d93ca9b26" Oct 30 16:27:31.016685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:31.016665 6400 scope.go:122] "RemoveContainer" containerID="287dd45d55fd2d5aea6a8ccb2844101819747e8229812cbb8afb1ddd7d3cdd87" Oct 30 16:27:31.023804 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:27:31.023784 6400 scope.go:122] "RemoveContainer" containerID="8b9e0a723a49c534b112e97c83f0933c27c9b33af7e05276efdd10acce8b300e" Oct 30 16:27:54.597653 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:27:54.597595 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:28:01.254327 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.254289 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn"] Oct 30 16:28:01.257359 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.257340 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.270318 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.270295 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 16:28:01.279468 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.279439 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 16:28:01.279717 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.279699 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"da112bd609bff87f7bacd2ba941225d7cff15c-bd73b2fc-dockercfg-ggpc7\"" Oct 30 16:28:01.281271 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.281251 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn"] Oct 30 16:28:01.313854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.313823 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th"] Oct 30 16:28:01.316751 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.316734 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.329118 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.329094 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e3d011e69d9573cabb8c359e5ac709eb5ac240-be78d160-dockercfg-knflj\"" Oct 30 16:28:01.337381 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.337359 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th"] Oct 30 16:28:01.346135 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.346112 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhjs4\" (UniqueName: \"kubernetes.io/projected/94e59209-b794-418b-b9bb-9f64f03cccbc-kube-api-access-lhjs4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.346241 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.346150 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fck4f\" (UniqueName: \"kubernetes.io/projected/c46681d3-e8e7-4631-833b-da0ae5666a05-kube-api-access-fck4f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.346241 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.346177 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.346316 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.346239 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.346316 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.346279 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.346382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.346316 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.409128 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.409099 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf"] Oct 30 16:28:01.412523 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.412496 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.426409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.426374 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3-b83c8cba-dockercfg-tmcz2\"" Oct 30 16:28:01.435115 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.435080 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf"] Oct 30 16:28:01.447198 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447167 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhjs4\" (UniqueName: \"kubernetes.io/projected/94e59209-b794-418b-b9bb-9f64f03cccbc-kube-api-access-lhjs4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.447370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447214 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fck4f\" (UniqueName: \"kubernetes.io/projected/c46681d3-e8e7-4631-833b-da0ae5666a05-kube-api-access-fck4f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.447370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447256 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.447370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447280 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.447370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447306 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.447370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447358 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.447665 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447395 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.447665 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447432 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.447665 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447458 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mv7dq\" (UniqueName: \"kubernetes.io/projected/6914eff7-ef28-49fd-a3d1-98fa273e5b19-kube-api-access-mv7dq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.447825 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447804 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.447865 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447820 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.447865 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447846 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.447925 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.447804 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.476146 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.476111 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhjs4\" (UniqueName: \"kubernetes.io/projected/94e59209-b794-418b-b9bb-9f64f03cccbc-kube-api-access-lhjs4\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.476871 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.476850 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fck4f\" (UniqueName: \"kubernetes.io/projected/c46681d3-e8e7-4631-833b-da0ae5666a05-kube-api-access-fck4f\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.540400 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.540295 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v"] Oct 30 16:28:01.543625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.543587 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.548231 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.548204 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.548349 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.548246 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.548349 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.548273 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mv7dq\" (UniqueName: \"kubernetes.io/projected/6914eff7-ef28-49fd-a3d1-98fa273e5b19-kube-api-access-mv7dq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.548564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.548546 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.548652 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.548587 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.556640 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.556617 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"6541d93aaee335d6227a8778ab4e840bbe7f52-f98fbcf8-dockercfg-ghn52\"" Oct 30 16:28:01.566543 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.566510 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v"] Oct 30 16:28:01.570822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.570793 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:01.572193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.572173 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mv7dq\" (UniqueName: \"kubernetes.io/projected/6914eff7-ef28-49fd-a3d1-98fa273e5b19-kube-api-access-mv7dq\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.630436 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.630401 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:01.649466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.649381 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.649675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.649504 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.649780 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.649706 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cspfx\" (UniqueName: \"kubernetes.io/projected/2182c592-7e85-40d0-ae6e-397d8027b0d2-kube-api-access-cspfx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.728951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.728920 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:01.751071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.751023 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cspfx\" (UniqueName: \"kubernetes.io/projected/2182c592-7e85-40d0-ae6e-397d8027b0d2-kube-api-access-cspfx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.751196 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.751162 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.751245 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.751218 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.751532 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.751512 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.751648 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.751550 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.772950 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.772910 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cspfx\" (UniqueName: \"kubernetes.io/projected/2182c592-7e85-40d0-ae6e-397d8027b0d2-kube-api-access-cspfx\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.858546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.858455 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:01.867009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.866986 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn"] Oct 30 16:28:01.870767 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:28:01.870738 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc46681d3_e8e7_4631_833b_da0ae5666a05.slice/crio-1c5e3aa0e3ca801f99f536f6833a6f3e034b7d8099c13e09475c7c492136ce62 WatchSource:0}: Error finding container 1c5e3aa0e3ca801f99f536f6833a6f3e034b7d8099c13e09475c7c492136ce62: Status 404 returned error can't find the container with id 1c5e3aa0e3ca801f99f536f6833a6f3e034b7d8099c13e09475c7c492136ce62 Oct 30 16:28:01.934698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:01.934667 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th"] Oct 30 16:28:01.938987 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:28:01.938953 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94e59209_b794_418b_b9bb_9f64f03cccbc.slice/crio-8a43de2aa56e8db0d84d6138d868fa22bc4cc7c387d6f3f76172ad03beeb4d38 WatchSource:0}: Error finding container 8a43de2aa56e8db0d84d6138d868fa22bc4cc7c387d6f3f76172ad03beeb4d38: Status 404 returned error can't find the container with id 8a43de2aa56e8db0d84d6138d868fa22bc4cc7c387d6f3f76172ad03beeb4d38 Oct 30 16:28:02.026428 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.026317 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf"] Oct 30 16:28:02.066424 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:28:02.066392 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6914eff7_ef28_49fd_a3d1_98fa273e5b19.slice/crio-0c6648e773a3b6ddcd9e41adc43cd03b2335e1e5806a6898e733b9c90720efbe WatchSource:0}: Error finding container 0c6648e773a3b6ddcd9e41adc43cd03b2335e1e5806a6898e733b9c90720efbe: Status 404 returned error can't find the container with id 0c6648e773a3b6ddcd9e41adc43cd03b2335e1e5806a6898e733b9c90720efbe Oct 30 16:28:02.162840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.162746 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v"] Oct 30 16:28:02.165870 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:28:02.165841 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2182c592_7e85_40d0_ae6e_397d8027b0d2.slice/crio-4b3b6cde868f74d19a01f5c2ebc6cb2b3ce4f92a0205d1f4e10b533d0425cb84 WatchSource:0}: Error finding container 4b3b6cde868f74d19a01f5c2ebc6cb2b3ce4f92a0205d1f4e10b533d0425cb84: Status 404 returned error can't find the container with id 4b3b6cde868f74d19a01f5c2ebc6cb2b3ce4f92a0205d1f4e10b533d0425cb84 Oct 30 16:28:02.775350 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.775317 6400 generic.go:356] "Generic (PLEG): container finished" podID="6914eff7-ef28-49fd-a3d1-98fa273e5b19" containerID="1504bd2ef932d408103e872b06b3f6a073fc51143f8c2fed4d96098866b05ebb" exitCode=0 Oct 30 16:28:02.775850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.775405 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" event={"ID":"6914eff7-ef28-49fd-a3d1-98fa273e5b19","Type":"ContainerDied","Data":"1504bd2ef932d408103e872b06b3f6a073fc51143f8c2fed4d96098866b05ebb"} Oct 30 16:28:02.775850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.775465 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" event={"ID":"6914eff7-ef28-49fd-a3d1-98fa273e5b19","Type":"ContainerStarted","Data":"0c6648e773a3b6ddcd9e41adc43cd03b2335e1e5806a6898e733b9c90720efbe"} Oct 30 16:28:02.776991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.776970 6400 generic.go:356] "Generic (PLEG): container finished" podID="2182c592-7e85-40d0-ae6e-397d8027b0d2" containerID="6da597cbcf71c2b75397126adca82f946d277e6ad8897792e027f032050881ec" exitCode=0 Oct 30 16:28:02.777075 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.777009 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" event={"ID":"2182c592-7e85-40d0-ae6e-397d8027b0d2","Type":"ContainerDied","Data":"6da597cbcf71c2b75397126adca82f946d277e6ad8897792e027f032050881ec"} Oct 30 16:28:02.777075 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.777060 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" event={"ID":"2182c592-7e85-40d0-ae6e-397d8027b0d2","Type":"ContainerStarted","Data":"4b3b6cde868f74d19a01f5c2ebc6cb2b3ce4f92a0205d1f4e10b533d0425cb84"} Oct 30 16:28:02.779065 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.778949 6400 generic.go:356] "Generic (PLEG): container finished" podID="94e59209-b794-418b-b9bb-9f64f03cccbc" containerID="5c2c360b10a0f71629f5b3a51cebe0337b43c012132a862e9c9df7dcc87f003a" exitCode=0 Oct 30 16:28:02.779065 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.779056 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" event={"ID":"94e59209-b794-418b-b9bb-9f64f03cccbc","Type":"ContainerDied","Data":"5c2c360b10a0f71629f5b3a51cebe0337b43c012132a862e9c9df7dcc87f003a"} Oct 30 16:28:02.779177 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.779099 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" event={"ID":"94e59209-b794-418b-b9bb-9f64f03cccbc","Type":"ContainerStarted","Data":"8a43de2aa56e8db0d84d6138d868fa22bc4cc7c387d6f3f76172ad03beeb4d38"} Oct 30 16:28:02.780595 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.780574 6400 generic.go:356] "Generic (PLEG): container finished" podID="c46681d3-e8e7-4631-833b-da0ae5666a05" containerID="fd58df7789e6271668266e2bf2046fd70c9e5877f8fe088158a742c50ed26e18" exitCode=0 Oct 30 16:28:02.780705 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.780626 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" event={"ID":"c46681d3-e8e7-4631-833b-da0ae5666a05","Type":"ContainerDied","Data":"fd58df7789e6271668266e2bf2046fd70c9e5877f8fe088158a742c50ed26e18"} Oct 30 16:28:02.780705 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:02.780675 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" event={"ID":"c46681d3-e8e7-4631-833b-da0ae5666a05","Type":"ContainerStarted","Data":"1c5e3aa0e3ca801f99f536f6833a6f3e034b7d8099c13e09475c7c492136ce62"} Oct 30 16:28:03.787579 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:03.787544 6400 generic.go:356] "Generic (PLEG): container finished" podID="94e59209-b794-418b-b9bb-9f64f03cccbc" containerID="fdc2c4c02d7c51c841a960bf93032f29564cf2e6fd4f40ff70abe63698559511" exitCode=0 Oct 30 16:28:03.788025 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:03.787634 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" event={"ID":"94e59209-b794-418b-b9bb-9f64f03cccbc","Type":"ContainerDied","Data":"fdc2c4c02d7c51c841a960bf93032f29564cf2e6fd4f40ff70abe63698559511"} Oct 30 16:28:03.789593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:03.789573 6400 generic.go:356] "Generic (PLEG): container finished" podID="c46681d3-e8e7-4631-833b-da0ae5666a05" containerID="4a025f0ff0ab426dbdf7a651ee518a313c8fd1d88121eeed3ea82079daefabb6" exitCode=0 Oct 30 16:28:03.789694 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:03.789641 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" event={"ID":"c46681d3-e8e7-4631-833b-da0ae5666a05","Type":"ContainerDied","Data":"4a025f0ff0ab426dbdf7a651ee518a313c8fd1d88121eeed3ea82079daefabb6"} Oct 30 16:28:03.791662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:03.791642 6400 generic.go:356] "Generic (PLEG): container finished" podID="6914eff7-ef28-49fd-a3d1-98fa273e5b19" containerID="1677e8e870929e8da0625d82a1dfbf7e173713c32c1f0f9eb27645729957e00a" exitCode=0 Oct 30 16:28:03.791797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:03.791713 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" event={"ID":"6914eff7-ef28-49fd-a3d1-98fa273e5b19","Type":"ContainerDied","Data":"1677e8e870929e8da0625d82a1dfbf7e173713c32c1f0f9eb27645729957e00a"} Oct 30 16:28:03.793630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:03.793595 6400 generic.go:356] "Generic (PLEG): container finished" podID="2182c592-7e85-40d0-ae6e-397d8027b0d2" containerID="fa7b5f761a3cd158582d410fa7e6ca826885ee471ad9af29d35ecbc4b82493a5" exitCode=0 Oct 30 16:28:03.793700 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:03.793666 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" event={"ID":"2182c592-7e85-40d0-ae6e-397d8027b0d2","Type":"ContainerDied","Data":"fa7b5f761a3cd158582d410fa7e6ca826885ee471ad9af29d35ecbc4b82493a5"} Oct 30 16:28:04.801557 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:04.801469 6400 generic.go:356] "Generic (PLEG): container finished" podID="c46681d3-e8e7-4631-833b-da0ae5666a05" containerID="70c8b55e6a786a22b1288fce966dc86c7c17e947ecfb554350616bd8bfadf93c" exitCode=0 Oct 30 16:28:04.801989 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:04.801550 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" event={"ID":"c46681d3-e8e7-4631-833b-da0ae5666a05","Type":"ContainerDied","Data":"70c8b55e6a786a22b1288fce966dc86c7c17e947ecfb554350616bd8bfadf93c"} Oct 30 16:28:04.803747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:04.803723 6400 generic.go:356] "Generic (PLEG): container finished" podID="6914eff7-ef28-49fd-a3d1-98fa273e5b19" containerID="6676cb76e5d0bdc6e68adafc78cdc56504ce68f8560e1f31ea4fad0bc0a05e9b" exitCode=0 Oct 30 16:28:04.803852 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:04.803807 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" event={"ID":"6914eff7-ef28-49fd-a3d1-98fa273e5b19","Type":"ContainerDied","Data":"6676cb76e5d0bdc6e68adafc78cdc56504ce68f8560e1f31ea4fad0bc0a05e9b"} Oct 30 16:28:04.805864 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:04.805842 6400 generic.go:356] "Generic (PLEG): container finished" podID="2182c592-7e85-40d0-ae6e-397d8027b0d2" containerID="138aa229866f09ef678d65c2a2f45a4da52c53c144fbbba251a161088deb7e96" exitCode=0 Oct 30 16:28:04.805977 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:04.805919 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" event={"ID":"2182c592-7e85-40d0-ae6e-397d8027b0d2","Type":"ContainerDied","Data":"138aa229866f09ef678d65c2a2f45a4da52c53c144fbbba251a161088deb7e96"} Oct 30 16:28:04.807884 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:04.807859 6400 generic.go:356] "Generic (PLEG): container finished" podID="94e59209-b794-418b-b9bb-9f64f03cccbc" containerID="2e854cdaec30a0126d46009e2cb7622ae1a9d70d978fd3883123c65b6f9664dd" exitCode=0 Oct 30 16:28:04.807980 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:04.807894 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" event={"ID":"94e59209-b794-418b-b9bb-9f64f03cccbc","Type":"ContainerDied","Data":"2e854cdaec30a0126d46009e2cb7622ae1a9d70d978fd3883123c65b6f9664dd"} Oct 30 16:28:05.975221 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:05.975191 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:06.019422 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.019396 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:06.022888 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.022866 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:06.026338 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.026317 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:06.092156 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092071 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-util\" (UniqueName: \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-util\") pod \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " Oct 30 16:28:06.092156 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092109 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94e59209-b794-418b-b9bb-9f64f03cccbc-kube-api-access-lhjs4\" (UniqueName: \"kubernetes.io/projected/94e59209-b794-418b-b9bb-9f64f03cccbc-kube-api-access-lhjs4\") pod \"94e59209-b794-418b-b9bb-9f64f03cccbc\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " Oct 30 16:28:06.092156 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092137 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2182c592-7e85-40d0-ae6e-397d8027b0d2-kube-api-access-cspfx\" (UniqueName: \"kubernetes.io/projected/2182c592-7e85-40d0-ae6e-397d8027b0d2-kube-api-access-cspfx\") pod \"2182c592-7e85-40d0-ae6e-397d8027b0d2\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " Oct 30 16:28:06.092442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092190 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46681d3-e8e7-4631-833b-da0ae5666a05-kube-api-access-fck4f\" (UniqueName: \"kubernetes.io/projected/c46681d3-e8e7-4631-833b-da0ae5666a05-kube-api-access-fck4f\") pod \"c46681d3-e8e7-4631-833b-da0ae5666a05\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " Oct 30 16:28:06.092442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092216 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-bundle\" (UniqueName: \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-bundle\") pod \"c46681d3-e8e7-4631-833b-da0ae5666a05\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " Oct 30 16:28:06.092442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092394 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-util\" (UniqueName: \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-util\") pod \"c46681d3-e8e7-4631-833b-da0ae5666a05\" (UID: \"c46681d3-e8e7-4631-833b-da0ae5666a05\") " Oct 30 16:28:06.092625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092464 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-bundle\" (UniqueName: \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-bundle\") pod \"94e59209-b794-418b-b9bb-9f64f03cccbc\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " Oct 30 16:28:06.092625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092494 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-bundle\" (UniqueName: \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-bundle\") pod \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " Oct 30 16:28:06.092625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092522 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-bundle\" (UniqueName: \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-bundle\") pod \"2182c592-7e85-40d0-ae6e-397d8027b0d2\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " Oct 30 16:28:06.092625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092561 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-util\" (UniqueName: \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-util\") pod \"2182c592-7e85-40d0-ae6e-397d8027b0d2\" (UID: \"2182c592-7e85-40d0-ae6e-397d8027b0d2\") " Oct 30 16:28:06.092843 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092649 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6914eff7-ef28-49fd-a3d1-98fa273e5b19-kube-api-access-mv7dq\" (UniqueName: \"kubernetes.io/projected/6914eff7-ef28-49fd-a3d1-98fa273e5b19-kube-api-access-mv7dq\") pod \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\" (UID: \"6914eff7-ef28-49fd-a3d1-98fa273e5b19\") " Oct 30 16:28:06.092843 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.092693 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-util\" (UniqueName: \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-util\") pod \"94e59209-b794-418b-b9bb-9f64f03cccbc\" (UID: \"94e59209-b794-418b-b9bb-9f64f03cccbc\") " Oct 30 16:28:06.093195 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.093145 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-bundle" pod "c46681d3-e8e7-4631-833b-da0ae5666a05" (UID: "c46681d3-e8e7-4631-833b-da0ae5666a05"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:06.093381 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.093208 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-bundle" pod "6914eff7-ef28-49fd-a3d1-98fa273e5b19" (UID: "6914eff7-ef28-49fd-a3d1-98fa273e5b19"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:06.094587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.093928 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-bundle" pod "94e59209-b794-418b-b9bb-9f64f03cccbc" (UID: "94e59209-b794-418b-b9bb-9f64f03cccbc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:06.095891 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.095258 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2182c592-7e85-40d0-ae6e-397d8027b0d2-kube-api-access-cspfx" pod "2182c592-7e85-40d0-ae6e-397d8027b0d2" (UID: "2182c592-7e85-40d0-ae6e-397d8027b0d2"). InnerVolumeSpecName "kube-api-access-cspfx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:06.095891 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.095465 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46681d3-e8e7-4631-833b-da0ae5666a05-kube-api-access-fck4f" pod "c46681d3-e8e7-4631-833b-da0ae5666a05" (UID: "c46681d3-e8e7-4631-833b-da0ae5666a05"). InnerVolumeSpecName "kube-api-access-fck4f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:06.095891 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.095640 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94e59209-b794-418b-b9bb-9f64f03cccbc-kube-api-access-lhjs4" pod "94e59209-b794-418b-b9bb-9f64f03cccbc" (UID: "94e59209-b794-418b-b9bb-9f64f03cccbc"). InnerVolumeSpecName "kube-api-access-lhjs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:06.098254 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.098198 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6914eff7-ef28-49fd-a3d1-98fa273e5b19-kube-api-access-mv7dq" pod "6914eff7-ef28-49fd-a3d1-98fa273e5b19" (UID: "6914eff7-ef28-49fd-a3d1-98fa273e5b19"). InnerVolumeSpecName "kube-api-access-mv7dq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:06.098693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.098396 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-bundle" pod "2182c592-7e85-40d0-ae6e-397d8027b0d2" (UID: "2182c592-7e85-40d0-ae6e-397d8027b0d2"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:06.101278 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.101255 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-util" pod "6914eff7-ef28-49fd-a3d1-98fa273e5b19" (UID: "6914eff7-ef28-49fd-a3d1-98fa273e5b19"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:06.102188 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.102129 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-util" pod "c46681d3-e8e7-4631-833b-da0ae5666a05" (UID: "c46681d3-e8e7-4631-833b-da0ae5666a05"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:06.102375 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.102353 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-util" pod "94e59209-b794-418b-b9bb-9f64f03cccbc" (UID: "94e59209-b794-418b-b9bb-9f64f03cccbc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:06.104673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.104650 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-util" pod "2182c592-7e85-40d0-ae6e-397d8027b0d2" (UID: "2182c592-7e85-40d0-ae6e-397d8027b0d2"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:06.194487 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194454 6400 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-bundle\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194487 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194483 6400 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-bundle\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194487 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194491 6400 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2182c592-7e85-40d0-ae6e-397d8027b0d2-util\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194500 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mv7dq\" (UniqueName: \"kubernetes.io/projected/6914eff7-ef28-49fd-a3d1-98fa273e5b19-kube-api-access-mv7dq\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194510 6400 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-util\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194519 6400 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6914eff7-ef28-49fd-a3d1-98fa273e5b19-util\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194526 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhjs4\" (UniqueName: \"kubernetes.io/projected/94e59209-b794-418b-b9bb-9f64f03cccbc-kube-api-access-lhjs4\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194535 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cspfx\" (UniqueName: \"kubernetes.io/projected/2182c592-7e85-40d0-ae6e-397d8027b0d2-kube-api-access-cspfx\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194543 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fck4f\" (UniqueName: \"kubernetes.io/projected/c46681d3-e8e7-4631-833b-da0ae5666a05-kube-api-access-fck4f\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194552 6400 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-bundle\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194560 6400 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c46681d3-e8e7-4631-833b-da0ae5666a05-util\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.194747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.194567 6400 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/94e59209-b794-418b-b9bb-9f64f03cccbc-bundle\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:06.824570 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.824540 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" Oct 30 16:28:06.824800 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.824537 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7xmmfn" event={"ID":"c46681d3-e8e7-4631-833b-da0ae5666a05","Type":"ContainerDied","Data":"1c5e3aa0e3ca801f99f536f6833a6f3e034b7d8099c13e09475c7c492136ce62"} Oct 30 16:28:06.824800 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.824694 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c5e3aa0e3ca801f99f536f6833a6f3e034b7d8099c13e09475c7c492136ce62" Oct 30 16:28:06.826533 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.826511 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" Oct 30 16:28:06.826681 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.826542 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c6hl2pf" event={"ID":"6914eff7-ef28-49fd-a3d1-98fa273e5b19","Type":"ContainerDied","Data":"0c6648e773a3b6ddcd9e41adc43cd03b2335e1e5806a6898e733b9c90720efbe"} Oct 30 16:28:06.826681 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.826582 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c6648e773a3b6ddcd9e41adc43cd03b2335e1e5806a6898e733b9c90720efbe" Oct 30 16:28:06.828500 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.828480 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" Oct 30 16:28:06.828586 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.828502 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847a28f6v" event={"ID":"2182c592-7e85-40d0-ae6e-397d8027b0d2","Type":"ContainerDied","Data":"4b3b6cde868f74d19a01f5c2ebc6cb2b3ce4f92a0205d1f4e10b533d0425cb84"} Oct 30 16:28:06.828586 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.828551 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b3b6cde868f74d19a01f5c2ebc6cb2b3ce4f92a0205d1f4e10b533d0425cb84" Oct 30 16:28:06.830594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.830572 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" event={"ID":"94e59209-b794-418b-b9bb-9f64f03cccbc","Type":"ContainerDied","Data":"8a43de2aa56e8db0d84d6138d868fa22bc4cc7c387d6f3f76172ad03beeb4d38"} Oct 30 16:28:06.830691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.830623 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a43de2aa56e8db0d84d6138d868fa22bc4cc7c387d6f3f76172ad03beeb4d38" Oct 30 16:28:06.830691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:06.830634 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4cjj7th" Oct 30 16:28:22.147233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147196 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs"] Oct 30 16:28:22.147818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147798 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6914eff7-ef28-49fd-a3d1-98fa273e5b19" containerName="util" Oct 30 16:28:22.147895 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147820 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6914eff7-ef28-49fd-a3d1-98fa273e5b19" containerName="util" Oct 30 16:28:22.147895 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147839 6400 cpu_manager.go:414] "containerMap: removing container" podUID="94e59209-b794-418b-b9bb-9f64f03cccbc" containerName="pull" Oct 30 16:28:22.147895 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147846 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e59209-b794-418b-b9bb-9f64f03cccbc" containerName="pull" Oct 30 16:28:22.147895 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147879 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c46681d3-e8e7-4631-833b-da0ae5666a05" containerName="extract" Oct 30 16:28:22.147895 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147886 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46681d3-e8e7-4631-833b-da0ae5666a05" containerName="extract" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147903 6400 cpu_manager.go:414] "containerMap: removing container" podUID="94e59209-b794-418b-b9bb-9f64f03cccbc" containerName="extract" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147910 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e59209-b794-418b-b9bb-9f64f03cccbc" containerName="extract" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147921 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c46681d3-e8e7-4631-833b-da0ae5666a05" containerName="pull" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147928 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46681d3-e8e7-4631-833b-da0ae5666a05" containerName="pull" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147937 6400 cpu_manager.go:414] "containerMap: removing container" podUID="2182c592-7e85-40d0-ae6e-397d8027b0d2" containerName="extract" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147944 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2182c592-7e85-40d0-ae6e-397d8027b0d2" containerName="extract" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147958 6400 cpu_manager.go:414] "containerMap: removing container" podUID="2182c592-7e85-40d0-ae6e-397d8027b0d2" containerName="pull" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147964 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2182c592-7e85-40d0-ae6e-397d8027b0d2" containerName="pull" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147977 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6914eff7-ef28-49fd-a3d1-98fa273e5b19" containerName="extract" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147984 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6914eff7-ef28-49fd-a3d1-98fa273e5b19" containerName="extract" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.147995 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c46681d3-e8e7-4631-833b-da0ae5666a05" containerName="util" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148002 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46681d3-e8e7-4631-833b-da0ae5666a05" containerName="util" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148011 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6914eff7-ef28-49fd-a3d1-98fa273e5b19" containerName="pull" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148018 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6914eff7-ef28-49fd-a3d1-98fa273e5b19" containerName="pull" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148033 6400 cpu_manager.go:414] "containerMap: removing container" podUID="2182c592-7e85-40d0-ae6e-397d8027b0d2" containerName="util" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148039 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2182c592-7e85-40d0-ae6e-397d8027b0d2" containerName="util" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148053 6400 cpu_manager.go:414] "containerMap: removing container" podUID="94e59209-b794-418b-b9bb-9f64f03cccbc" containerName="util" Oct 30 16:28:22.148142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148060 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94e59209-b794-418b-b9bb-9f64f03cccbc" containerName="util" Oct 30 16:28:22.148962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148153 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46681d3-e8e7-4631-833b-da0ae5666a05" containerName="extract" Oct 30 16:28:22.148962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148167 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="94e59209-b794-418b-b9bb-9f64f03cccbc" containerName="extract" Oct 30 16:28:22.148962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148177 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="2182c592-7e85-40d0-ae6e-397d8027b0d2" containerName="extract" Oct 30 16:28:22.148962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.148187 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6914eff7-ef28-49fd-a3d1-98fa273e5b19" containerName="extract" Oct 30 16:28:22.150596 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.150555 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.163436 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.163410 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" Oct 30 16:28:22.168320 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.168302 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 30 16:28:22.168405 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.168324 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 30 16:28:22.172939 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.172921 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" Oct 30 16:28:22.173052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.173030 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vtlxz\"" Oct 30 16:28:22.175903 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.175880 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs"] Oct 30 16:28:22.261707 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.261676 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/79795cb8-7a9c-4aa0-8bd1-1dec782af242-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6xzjs\" (UID: \"79795cb8-7a9c-4aa0-8bd1-1dec782af242\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.261889 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.261800 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtsfq\" (UniqueName: \"kubernetes.io/projected/79795cb8-7a9c-4aa0-8bd1-1dec782af242-kube-api-access-wtsfq\") pod \"kuadrant-console-plugin-66c4fb575d-6xzjs\" (UID: \"79795cb8-7a9c-4aa0-8bd1-1dec782af242\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.261889 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.261849 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/79795cb8-7a9c-4aa0-8bd1-1dec782af242-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6xzjs\" (UID: \"79795cb8-7a9c-4aa0-8bd1-1dec782af242\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.362661 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.362622 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/79795cb8-7a9c-4aa0-8bd1-1dec782af242-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6xzjs\" (UID: \"79795cb8-7a9c-4aa0-8bd1-1dec782af242\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.362874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.362679 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/79795cb8-7a9c-4aa0-8bd1-1dec782af242-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6xzjs\" (UID: \"79795cb8-7a9c-4aa0-8bd1-1dec782af242\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.362874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.362747 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtsfq\" (UniqueName: \"kubernetes.io/projected/79795cb8-7a9c-4aa0-8bd1-1dec782af242-kube-api-access-wtsfq\") pod \"kuadrant-console-plugin-66c4fb575d-6xzjs\" (UID: \"79795cb8-7a9c-4aa0-8bd1-1dec782af242\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.363439 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.363417 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/79795cb8-7a9c-4aa0-8bd1-1dec782af242-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-6xzjs\" (UID: \"79795cb8-7a9c-4aa0-8bd1-1dec782af242\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.365247 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.365224 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/79795cb8-7a9c-4aa0-8bd1-1dec782af242-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-6xzjs\" (UID: \"79795cb8-7a9c-4aa0-8bd1-1dec782af242\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.388560 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.388534 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtsfq\" (UniqueName: \"kubernetes.io/projected/79795cb8-7a9c-4aa0-8bd1-1dec782af242-kube-api-access-wtsfq\") pod \"kuadrant-console-plugin-66c4fb575d-6xzjs\" (UID: \"79795cb8-7a9c-4aa0-8bd1-1dec782af242\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.462166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.462086 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" Oct 30 16:28:22.757028 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.756999 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs"] Oct 30 16:28:22.759528 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:28:22.759493 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod79795cb8_7a9c_4aa0_8bd1_1dec782af242.slice/crio-3ec6331634c74982a14b5296d0a704e48b334e14f6d3d5fb2c15b175e992dbae WatchSource:0}: Error finding container 3ec6331634c74982a14b5296d0a704e48b334e14f6d3d5fb2c15b175e992dbae: Status 404 returned error can't find the container with id 3ec6331634c74982a14b5296d0a704e48b334e14f6d3d5fb2c15b175e992dbae Oct 30 16:28:22.905311 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:22.905279 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" event={"ID":"79795cb8-7a9c-4aa0-8bd1-1dec782af242","Type":"ContainerStarted","Data":"3ec6331634c74982a14b5296d0a704e48b334e14f6d3d5fb2c15b175e992dbae"} Oct 30 16:28:24.601342 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:28:24.601301 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:28:27.936313 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:27.936275 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" event={"ID":"79795cb8-7a9c-4aa0-8bd1-1dec782af242","Type":"ContainerStarted","Data":"d57ebc857befd8b377cf33b70a848681e01def6e46ea7e4871cea1efdaaeb541"} Oct 30 16:28:27.974973 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:27.974916 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-6xzjs" podStartSLOduration=1.66718682 podStartE2EDuration="5.974902329s" totalImagesPullingTime="4.307715509s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:27.972771348 +0000 UTC m=+417.123052336" watchObservedRunningTime="2026-10-30 16:28:27.974902329 +0000 UTC m=+417.125183313" Oct 30 16:28:32.311698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.311668 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk"] Oct 30 16:28:32.315084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.315064 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:32.340219 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.340194 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk"] Oct 30 16:28:32.426469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.426437 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-gwddn\"" Oct 30 16:28:32.461902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.461863 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-5ksgk\" (UID: \"dc6ab118-6fe2-4025-8856-0fbe0edd1b13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:32.462094 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.461979 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nntjz\" (UniqueName: \"kubernetes.io/projected/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-kube-api-access-nntjz\") pod \"kuadrant-operator-controller-manager-6b76784d48-5ksgk\" (UID: \"dc6ab118-6fe2-4025-8856-0fbe0edd1b13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:32.563235 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.563131 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-5ksgk\" (UID: \"dc6ab118-6fe2-4025-8856-0fbe0edd1b13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:32.563235 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.563219 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nntjz\" (UniqueName: \"kubernetes.io/projected/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-kube-api-access-nntjz\") pod \"kuadrant-operator-controller-manager-6b76784d48-5ksgk\" (UID: \"dc6ab118-6fe2-4025-8856-0fbe0edd1b13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:32.563533 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.563507 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-5ksgk\" (UID: \"dc6ab118-6fe2-4025-8856-0fbe0edd1b13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:32.587810 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.587773 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nntjz\" (UniqueName: \"kubernetes.io/projected/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-kube-api-access-nntjz\") pod \"kuadrant-operator-controller-manager-6b76784d48-5ksgk\" (UID: \"dc6ab118-6fe2-4025-8856-0fbe0edd1b13\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:32.628677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.628639 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:32.949740 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.949593 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk"] Oct 30 16:28:32.952556 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:28:32.952528 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc6ab118_6fe2_4025_8856_0fbe0edd1b13.slice/crio-3722e0a1577061180280993bedf0ccaf02de9b2a2fd14d29e1190002a3ebc997 WatchSource:0}: Error finding container 3722e0a1577061180280993bedf0ccaf02de9b2a2fd14d29e1190002a3ebc997: Status 404 returned error can't find the container with id 3722e0a1577061180280993bedf0ccaf02de9b2a2fd14d29e1190002a3ebc997 Oct 30 16:28:32.971880 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:32.971851 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" event={"ID":"dc6ab118-6fe2-4025-8856-0fbe0edd1b13","Type":"ContainerStarted","Data":"3722e0a1577061180280993bedf0ccaf02de9b2a2fd14d29e1190002a3ebc997"} Oct 30 16:28:34.616940 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.616841 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk"] Oct 30 16:28:34.617938 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.617902 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk"] Oct 30 16:28:34.629537 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.629511 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7"] Oct 30 16:28:34.637080 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.636876 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:28:34.664368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.664340 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7"] Oct 30 16:28:34.789327 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.789295 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjljq\" (UniqueName: \"kubernetes.io/projected/39a68b03-0029-4d07-8c60-14f84311bd23-kube-api-access-bjljq\") pod \"kuadrant-operator-controller-manager-6b76784d48-xgjm7\" (UID: \"39a68b03-0029-4d07-8c60-14f84311bd23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:28:34.789516 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.789382 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39a68b03-0029-4d07-8c60-14f84311bd23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-xgjm7\" (UID: \"39a68b03-0029-4d07-8c60-14f84311bd23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:28:34.891320 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.890848 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjljq\" (UniqueName: \"kubernetes.io/projected/39a68b03-0029-4d07-8c60-14f84311bd23-kube-api-access-bjljq\") pod \"kuadrant-operator-controller-manager-6b76784d48-xgjm7\" (UID: \"39a68b03-0029-4d07-8c60-14f84311bd23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:28:34.891320 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.891089 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39a68b03-0029-4d07-8c60-14f84311bd23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-xgjm7\" (UID: \"39a68b03-0029-4d07-8c60-14f84311bd23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:28:34.891576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.891504 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39a68b03-0029-4d07-8c60-14f84311bd23-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6b76784d48-xgjm7\" (UID: \"39a68b03-0029-4d07-8c60-14f84311bd23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:28:34.913241 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.913205 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjljq\" (UniqueName: \"kubernetes.io/projected/39a68b03-0029-4d07-8c60-14f84311bd23-kube-api-access-bjljq\") pod \"kuadrant-operator-controller-manager-6b76784d48-xgjm7\" (UID: \"39a68b03-0029-4d07-8c60-14f84311bd23\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:28:34.956761 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:34.956495 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:28:35.322377 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:35.322343 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7"] Oct 30 16:28:35.336286 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:28:35.336256 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39a68b03_0029_4d07_8c60_14f84311bd23.slice/crio-4b4add002d5c5c12f9c909c0f1505fcad8c77555ab87e7fb60b6e17be4a0dcce WatchSource:0}: Error finding container 4b4add002d5c5c12f9c909c0f1505fcad8c77555ab87e7fb60b6e17be4a0dcce: Status 404 returned error can't find the container with id 4b4add002d5c5c12f9c909c0f1505fcad8c77555ab87e7fb60b6e17be4a0dcce Oct 30 16:28:35.992029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:35.991987 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" event={"ID":"39a68b03-0029-4d07-8c60-14f84311bd23","Type":"ContainerStarted","Data":"4b4add002d5c5c12f9c909c0f1505fcad8c77555ab87e7fb60b6e17be4a0dcce"} Oct 30 16:28:38.004539 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.004501 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" event={"ID":"39a68b03-0029-4d07-8c60-14f84311bd23","Type":"ContainerStarted","Data":"4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c"} Oct 30 16:28:38.005028 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.004577 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:28:38.006482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.006455 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" podUID="dc6ab118-6fe2-4025-8856-0fbe0edd1b13" containerName="manager" containerID="cri-o://d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3" gracePeriod=2 Oct 30 16:28:38.068176 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.068129 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" podStartSLOduration=1.6976540020000002 podStartE2EDuration="4.068114659s" totalImagesPullingTime="2.370460657s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:28:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:28:38.05796196 +0000 UTC m=+427.208242948" watchObservedRunningTime="2026-10-30 16:28:38.068114659 +0000 UTC m=+427.218395646" Oct 30 16:28:38.268297 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.268243 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:38.279302 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:28:38.279264 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-5ksgk\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="dc6ab118-6fe2-4025-8856-0fbe0edd1b13" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:38.330451 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.330408 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-extensions-socket-volume\") pod \"dc6ab118-6fe2-4025-8856-0fbe0edd1b13\" (UID: \"dc6ab118-6fe2-4025-8856-0fbe0edd1b13\") " Oct 30 16:28:38.330672 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.330488 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-kube-api-access-nntjz\" (UniqueName: \"kubernetes.io/projected/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-kube-api-access-nntjz\") pod \"dc6ab118-6fe2-4025-8856-0fbe0edd1b13\" (UID: \"dc6ab118-6fe2-4025-8856-0fbe0edd1b13\") " Oct 30 16:28:38.330743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.330709 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-extensions-socket-volume" pod "dc6ab118-6fe2-4025-8856-0fbe0edd1b13" (UID: "dc6ab118-6fe2-4025-8856-0fbe0edd1b13"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:28:38.332817 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.332781 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-kube-api-access-nntjz" pod "dc6ab118-6fe2-4025-8856-0fbe0edd1b13" (UID: "dc6ab118-6fe2-4025-8856-0fbe0edd1b13"). InnerVolumeSpecName "kube-api-access-nntjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:28:38.431301 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.431269 6400 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-extensions-socket-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:38.431301 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.431295 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nntjz\" (UniqueName: \"kubernetes.io/projected/dc6ab118-6fe2-4025-8856-0fbe0edd1b13-kube-api-access-nntjz\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:28:38.965796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:38.965760 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc6ab118-6fe2-4025-8856-0fbe0edd1b13" path="/var/lib/kubelet/pods/dc6ab118-6fe2-4025-8856-0fbe0edd1b13/volumes" Oct 30 16:28:39.012617 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:39.012546 6400 generic.go:356] "Generic (PLEG): container finished" podID="dc6ab118-6fe2-4025-8856-0fbe0edd1b13" containerID="d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3" exitCode=2 Oct 30 16:28:39.012617 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:39.012595 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:39.013089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:39.012648 6400 scope.go:122] "RemoveContainer" containerID="d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3" Oct 30 16:28:39.021512 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:39.021488 6400 scope.go:122] "RemoveContainer" containerID="d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3" Oct 30 16:28:39.021810 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:28:39.021788 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3\": container with ID starting with d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3 not found: ID does not exist" containerID="d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3" Oct 30 16:28:39.021880 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:39.021823 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3"} err="failed to get container status \"d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3\": rpc error: code = NotFound desc = could not find container \"d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3\": container with ID starting with d5fbc7d460c687225a82d123edf9cc25a4bb1e87c9550ea03f7e5c4b121f40f3 not found: ID does not exist" Oct 30 16:28:39.025982 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:28:39.025954 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-5ksgk\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="dc6ab118-6fe2-4025-8856-0fbe0edd1b13" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:40.973912 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:28:40.973875 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6b76784d48-5ksgk\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="dc6ab118-6fe2-4025-8856-0fbe0edd1b13" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-5ksgk" Oct 30 16:28:49.014811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:28:49.014781 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:28:54.598364 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:28:54.598326 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:29:05.214563 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:05.214527 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7"] Oct 30 16:29:05.215133 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:05.214761 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" podUID="39a68b03-0029-4d07-8c60-14f84311bd23" containerName="manager" containerID="cri-o://4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c" gracePeriod=10 Oct 30 16:29:05.472243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:05.472162 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:29:05.579567 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:05.579538 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/39a68b03-0029-4d07-8c60-14f84311bd23-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39a68b03-0029-4d07-8c60-14f84311bd23-extensions-socket-volume\") pod \"39a68b03-0029-4d07-8c60-14f84311bd23\" (UID: \"39a68b03-0029-4d07-8c60-14f84311bd23\") " Oct 30 16:29:05.579759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:05.579719 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39a68b03-0029-4d07-8c60-14f84311bd23-kube-api-access-bjljq\" (UniqueName: \"kubernetes.io/projected/39a68b03-0029-4d07-8c60-14f84311bd23-kube-api-access-bjljq\") pod \"39a68b03-0029-4d07-8c60-14f84311bd23\" (UID: \"39a68b03-0029-4d07-8c60-14f84311bd23\") " Oct 30 16:29:05.580108 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:05.580087 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/39a68b03-0029-4d07-8c60-14f84311bd23-extensions-socket-volume" pod "39a68b03-0029-4d07-8c60-14f84311bd23" (UID: "39a68b03-0029-4d07-8c60-14f84311bd23"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:29:05.581980 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:05.581950 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39a68b03-0029-4d07-8c60-14f84311bd23-kube-api-access-bjljq" pod "39a68b03-0029-4d07-8c60-14f84311bd23" (UID: "39a68b03-0029-4d07-8c60-14f84311bd23"). InnerVolumeSpecName "kube-api-access-bjljq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:05.681182 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:05.681139 6400 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39a68b03-0029-4d07-8c60-14f84311bd23-extensions-socket-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:29:05.681182 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:05.681178 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjljq\" (UniqueName: \"kubernetes.io/projected/39a68b03-0029-4d07-8c60-14f84311bd23-kube-api-access-bjljq\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:29:06.147564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:06.147530 6400 generic.go:356] "Generic (PLEG): container finished" podID="39a68b03-0029-4d07-8c60-14f84311bd23" containerID="4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c" exitCode=0 Oct 30 16:29:06.147759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:06.147575 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" Oct 30 16:29:06.147759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:06.147633 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" event={"ID":"39a68b03-0029-4d07-8c60-14f84311bd23","Type":"ContainerDied","Data":"4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c"} Oct 30 16:29:06.147759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:06.147689 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7" event={"ID":"39a68b03-0029-4d07-8c60-14f84311bd23","Type":"ContainerDied","Data":"4b4add002d5c5c12f9c909c0f1505fcad8c77555ab87e7fb60b6e17be4a0dcce"} Oct 30 16:29:06.147759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:06.147713 6400 scope.go:122] "RemoveContainer" containerID="4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c" Oct 30 16:29:06.161493 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:06.161468 6400 scope.go:122] "RemoveContainer" containerID="4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c" Oct 30 16:29:06.161801 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:29:06.161782 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c\": container with ID starting with 4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c not found: ID does not exist" containerID="4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c" Oct 30 16:29:06.161865 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:06.161812 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c"} err="failed to get container status \"4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c\": rpc error: code = NotFound desc = could not find container \"4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c\": container with ID starting with 4346aad0b4517329097ce6b4dab9aca4016a3c195087c2555b07027c72860e0c not found: ID does not exist" Oct 30 16:29:06.202816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:06.202788 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7"] Oct 30 16:29:06.209496 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:06.209477 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6b76784d48-xgjm7"] Oct 30 16:29:06.965310 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:06.965276 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39a68b03-0029-4d07-8c60-14f84311bd23" path="/var/lib/kubelet/pods/39a68b03-0029-4d07-8c60-14f84311bd23/volumes" Oct 30 16:29:16.494240 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.494200 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw"] Oct 30 16:29:16.494823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.494801 6400 cpu_manager.go:414] "containerMap: removing container" podUID="39a68b03-0029-4d07-8c60-14f84311bd23" containerName="manager" Oct 30 16:29:16.494984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.494964 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39a68b03-0029-4d07-8c60-14f84311bd23" containerName="manager" Oct 30 16:29:16.495095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.494991 6400 cpu_manager.go:414] "containerMap: removing container" podUID="dc6ab118-6fe2-4025-8856-0fbe0edd1b13" containerName="manager" Oct 30 16:29:16.495095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.495000 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc6ab118-6fe2-4025-8856-0fbe0edd1b13" containerName="manager" Oct 30 16:29:16.495175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.495123 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="39a68b03-0029-4d07-8c60-14f84311bd23" containerName="manager" Oct 30 16:29:16.495175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.495133 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc6ab118-6fe2-4025-8856-0fbe0edd1b13" containerName="manager" Oct 30 16:29:16.498206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.498190 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.512958 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.512918 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nztr5\"" Oct 30 16:29:16.524850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.524824 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw"] Oct 30 16:29:16.685984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.685957 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.686175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.685996 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.686175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.686017 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.686175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.686143 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.686175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.686161 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.686390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.686189 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.686390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.686336 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.686496 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.686398 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.686496 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.686441 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9x94\" (UniqueName: \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-kube-api-access-t9x94\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.787587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.787510 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.787587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.787549 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9x94\" (UniqueName: \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-kube-api-access-t9x94\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.787587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.787586 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.787901 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.787645 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.787901 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.787807 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.787901 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.787887 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.788019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.787915 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.788073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.788013 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.788129 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.788066 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.788129 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.788075 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.788225 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.788199 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.788286 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.788217 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.788541 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.788522 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.788704 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.788684 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.790255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.790233 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.790344 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.790261 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.808719 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.808698 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9x94\" (UniqueName: \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-kube-api-access-t9x94\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.808823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.808745 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-xpkrw\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:16.810575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:16.810554 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:17.161811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:17.161789 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw"] Oct 30 16:29:17.163620 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:29:17.163579 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0349924c_b2b5_4a40_b6aa_9206c8046bc8.slice/crio-c7ff0f80ccc5718eb31b3b28f91058dfd9636edcdb8177d31bcf982b14836e2f WatchSource:0}: Error finding container c7ff0f80ccc5718eb31b3b28f91058dfd9636edcdb8177d31bcf982b14836e2f: Status 404 returned error can't find the container with id c7ff0f80ccc5718eb31b3b28f91058dfd9636edcdb8177d31bcf982b14836e2f Oct 30 16:29:17.165923 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:17.165888 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:29:17.166020 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:17.165958 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:29:17.166020 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:17.165988 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:29:17.205378 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:17.205354 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" event={"ID":"0349924c-b2b5-4a40-b6aa-9206c8046bc8","Type":"ContainerStarted","Data":"c7ff0f80ccc5718eb31b3b28f91058dfd9636edcdb8177d31bcf982b14836e2f"} Oct 30 16:29:18.212123 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:18.212082 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" event={"ID":"0349924c-b2b5-4a40-b6aa-9206c8046bc8","Type":"ContainerStarted","Data":"46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277"} Oct 30 16:29:18.265331 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:18.265273 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" podStartSLOduration=2.265258713 podStartE2EDuration="2.265258713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:18.26333245 +0000 UTC m=+467.413613437" watchObservedRunningTime="2026-10-30 16:29:18.265258713 +0000 UTC m=+467.415539701" Oct 30 16:29:18.811008 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:18.810974 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:18.816061 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:18.816035 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:19.217559 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:19.217477 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:19.219007 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:19.218988 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:29:24.586239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:24.586206 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nb7ld"] Oct 30 16:29:24.589454 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:24.589433 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nb7ld" Oct 30 16:29:24.599195 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:29:24.599162 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:29:24.603909 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:24.603871 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rsnhr\"" Oct 30 16:29:24.613736 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:24.613713 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nb7ld"] Oct 30 16:29:24.645584 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:24.645554 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkbs4\" (UniqueName: \"kubernetes.io/projected/a3cbbacc-fe37-447b-b446-c88b60e3fe53-kube-api-access-bkbs4\") pod \"authorino-57f7f8659f-nb7ld\" (UID: \"a3cbbacc-fe37-447b-b446-c88b60e3fe53\") " pod="kuadrant-system/authorino-57f7f8659f-nb7ld" Oct 30 16:29:24.746101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:24.746067 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkbs4\" (UniqueName: \"kubernetes.io/projected/a3cbbacc-fe37-447b-b446-c88b60e3fe53-kube-api-access-bkbs4\") pod \"authorino-57f7f8659f-nb7ld\" (UID: \"a3cbbacc-fe37-447b-b446-c88b60e3fe53\") " pod="kuadrant-system/authorino-57f7f8659f-nb7ld" Oct 30 16:29:24.767396 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:24.767366 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkbs4\" (UniqueName: \"kubernetes.io/projected/a3cbbacc-fe37-447b-b446-c88b60e3fe53-kube-api-access-bkbs4\") pod \"authorino-57f7f8659f-nb7ld\" (UID: \"a3cbbacc-fe37-447b-b446-c88b60e3fe53\") " pod="kuadrant-system/authorino-57f7f8659f-nb7ld" Oct 30 16:29:24.903765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:24.903682 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nb7ld" Oct 30 16:29:25.207042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:25.207013 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nb7ld"] Oct 30 16:29:25.209483 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:29:25.209449 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3cbbacc_fe37_447b_b446_c88b60e3fe53.slice/crio-84a55ee0302498e84becce91bface68ac0c53197fe16f5a08cf8311d945ef955 WatchSource:0}: Error finding container 84a55ee0302498e84becce91bface68ac0c53197fe16f5a08cf8311d945ef955: Status 404 returned error can't find the container with id 84a55ee0302498e84becce91bface68ac0c53197fe16f5a08cf8311d945ef955 Oct 30 16:29:25.210846 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:25.210824 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:29:25.245817 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:25.245784 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nb7ld" event={"ID":"a3cbbacc-fe37-447b-b446-c88b60e3fe53","Type":"ContainerStarted","Data":"84a55ee0302498e84becce91bface68ac0c53197fe16f5a08cf8311d945ef955"} Oct 30 16:29:27.258528 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:27.258493 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nb7ld" event={"ID":"a3cbbacc-fe37-447b-b446-c88b60e3fe53","Type":"ContainerStarted","Data":"31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a"} Oct 30 16:29:27.295722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:27.295671 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-nb7ld" podStartSLOduration=1.402428246 podStartE2EDuration="3.295656532s" totalImagesPullingTime="1.893228286s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:27.293785135 +0000 UTC m=+476.444066128" watchObservedRunningTime="2026-10-30 16:29:27.295656532 +0000 UTC m=+476.445937519" Oct 30 16:29:28.068120 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:28.068089 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nb7ld"] Oct 30 16:29:29.269022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:29.268971 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-nb7ld" podUID="a3cbbacc-fe37-447b-b446-c88b60e3fe53" containerName="authorino" containerID="cri-o://31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a" gracePeriod=30 Oct 30 16:29:29.511266 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:29.511239 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nb7ld" Oct 30 16:29:29.587729 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:29.587654 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3cbbacc-fe37-447b-b446-c88b60e3fe53-kube-api-access-bkbs4\" (UniqueName: \"kubernetes.io/projected/a3cbbacc-fe37-447b-b446-c88b60e3fe53-kube-api-access-bkbs4\") pod \"a3cbbacc-fe37-447b-b446-c88b60e3fe53\" (UID: \"a3cbbacc-fe37-447b-b446-c88b60e3fe53\") " Oct 30 16:29:29.589882 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:29.589858 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3cbbacc-fe37-447b-b446-c88b60e3fe53-kube-api-access-bkbs4" pod "a3cbbacc-fe37-447b-b446-c88b60e3fe53" (UID: "a3cbbacc-fe37-447b-b446-c88b60e3fe53"). InnerVolumeSpecName "kube-api-access-bkbs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:29.688531 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:29.688494 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkbs4\" (UniqueName: \"kubernetes.io/projected/a3cbbacc-fe37-447b-b446-c88b60e3fe53-kube-api-access-bkbs4\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:29:30.280671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:30.280631 6400 generic.go:356] "Generic (PLEG): container finished" podID="a3cbbacc-fe37-447b-b446-c88b60e3fe53" containerID="31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a" exitCode=0 Oct 30 16:29:30.280671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:30.280663 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-nb7ld" Oct 30 16:29:30.281238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:30.280664 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nb7ld" event={"ID":"a3cbbacc-fe37-447b-b446-c88b60e3fe53","Type":"ContainerDied","Data":"31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a"} Oct 30 16:29:30.281238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:30.280770 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-nb7ld" event={"ID":"a3cbbacc-fe37-447b-b446-c88b60e3fe53","Type":"ContainerDied","Data":"84a55ee0302498e84becce91bface68ac0c53197fe16f5a08cf8311d945ef955"} Oct 30 16:29:30.281238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:30.280787 6400 scope.go:122] "RemoveContainer" containerID="31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a" Oct 30 16:29:30.289961 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:30.289941 6400 scope.go:122] "RemoveContainer" containerID="31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a" Oct 30 16:29:30.290239 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:29:30.290219 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a\": container with ID starting with 31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a not found: ID does not exist" containerID="31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a" Oct 30 16:29:30.290303 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:30.290253 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a"} err="failed to get container status \"31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a\": rpc error: code = NotFound desc = could not find container \"31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a\": container with ID starting with 31e2d255d16ec496d25426c10c341052cd53471c828670a8578215769a1bb84a not found: ID does not exist" Oct 30 16:29:30.325441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:30.325415 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nb7ld"] Oct 30 16:29:30.329186 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:30.329145 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-nb7ld"] Oct 30 16:29:30.965577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:30.965544 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3cbbacc-fe37-447b-b446-c88b60e3fe53" path="/var/lib/kubelet/pods/a3cbbacc-fe37-447b-b446-c88b60e3fe53/volumes" Oct 30 16:29:42.584368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.584331 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm"] Oct 30 16:29:42.584918 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.584840 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a3cbbacc-fe37-447b-b446-c88b60e3fe53" containerName="authorino" Oct 30 16:29:42.584918 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.584855 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3cbbacc-fe37-447b-b446-c88b60e3fe53" containerName="authorino" Oct 30 16:29:42.584994 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.584926 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3cbbacc-fe37-447b-b446-c88b60e3fe53" containerName="authorino" Oct 30 16:29:42.587886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.587871 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" Oct 30 16:29:42.601774 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.601753 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 16:29:42.610863 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.610845 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xn9wb\"" Oct 30 16:29:42.611175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.611157 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 16:29:42.611216 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.611170 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 16:29:42.624518 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.624495 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm"] Oct 30 16:29:42.702977 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.702948 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c4b56d9a-9f54-417a-89d0-bde1880c0dda-data\") pod \"postgres-7cdbf6dc44-rkmxm\" (UID: \"c4b56d9a-9f54-417a-89d0-bde1880c0dda\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" Oct 30 16:29:42.703138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.702989 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4t5m\" (UniqueName: \"kubernetes.io/projected/c4b56d9a-9f54-417a-89d0-bde1880c0dda-kube-api-access-z4t5m\") pod \"postgres-7cdbf6dc44-rkmxm\" (UID: \"c4b56d9a-9f54-417a-89d0-bde1880c0dda\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" Oct 30 16:29:42.804001 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.803968 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4t5m\" (UniqueName: \"kubernetes.io/projected/c4b56d9a-9f54-417a-89d0-bde1880c0dda-kube-api-access-z4t5m\") pod \"postgres-7cdbf6dc44-rkmxm\" (UID: \"c4b56d9a-9f54-417a-89d0-bde1880c0dda\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" Oct 30 16:29:42.804178 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.804083 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c4b56d9a-9f54-417a-89d0-bde1880c0dda-data\") pod \"postgres-7cdbf6dc44-rkmxm\" (UID: \"c4b56d9a-9f54-417a-89d0-bde1880c0dda\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" Oct 30 16:29:42.804419 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.804403 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/c4b56d9a-9f54-417a-89d0-bde1880c0dda-data\") pod \"postgres-7cdbf6dc44-rkmxm\" (UID: \"c4b56d9a-9f54-417a-89d0-bde1880c0dda\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" Oct 30 16:29:42.835715 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.835647 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4t5m\" (UniqueName: \"kubernetes.io/projected/c4b56d9a-9f54-417a-89d0-bde1880c0dda-kube-api-access-z4t5m\") pod \"postgres-7cdbf6dc44-rkmxm\" (UID: \"c4b56d9a-9f54-417a-89d0-bde1880c0dda\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" Oct 30 16:29:42.901644 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:42.901589 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" Oct 30 16:29:43.199398 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:43.199349 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm"] Oct 30 16:29:43.201718 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:29:43.201684 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4b56d9a_9f54_417a_89d0_bde1880c0dda.slice/crio-f40ee4b1dccb308e8fbd13f51d89a0c125e60221c9e27279b8835d2387b4a323 WatchSource:0}: Error finding container f40ee4b1dccb308e8fbd13f51d89a0c125e60221c9e27279b8835d2387b4a323: Status 404 returned error can't find the container with id f40ee4b1dccb308e8fbd13f51d89a0c125e60221c9e27279b8835d2387b4a323 Oct 30 16:29:43.341539 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:43.341502 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" event={"ID":"c4b56d9a-9f54-417a-89d0-bde1880c0dda","Type":"ContainerStarted","Data":"f40ee4b1dccb308e8fbd13f51d89a0c125e60221c9e27279b8835d2387b4a323"} Oct 30 16:29:48.374255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:48.374159 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" event={"ID":"c4b56d9a-9f54-417a-89d0-bde1880c0dda","Type":"ContainerStarted","Data":"467cdf69742ded7a8334b24fe6ca06554570c96d9235b70dacbc5205dadc9d08"} Oct 30 16:29:48.374686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:48.374254 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" Oct 30 16:29:48.409961 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:48.409902 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" podStartSLOduration=1.566208333 podStartE2EDuration="6.409886399s" totalImagesPullingTime="4.843678066s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:48.4091789 +0000 UTC m=+497.559459888" watchObservedRunningTime="2026-10-30 16:29:48.409886399 +0000 UTC m=+497.560167388" Oct 30 16:29:54.411834 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:54.411804 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-rkmxm" Oct 30 16:29:54.596287 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:29:54.596248 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:29:55.469345 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.469310 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-xzvfg"] Oct 30 16:29:55.475678 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.475658 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" Oct 30 16:29:55.493999 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.493977 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-xzvfg"] Oct 30 16:29:55.501830 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.501803 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rsnhr\"" Oct 30 16:29:55.501947 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.501886 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" Oct 30 16:29:55.627714 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.627679 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g465\" (UniqueName: \"kubernetes.io/projected/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-kube-api-access-2g465\") pod \"authorino-6889cbd8c4-xzvfg\" (UID: \"c2e63f29-7465-4a8d-a96f-0deb3c76b3a3\") " pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" Oct 30 16:29:55.627886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.627822 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-tls-cert\") pod \"authorino-6889cbd8c4-xzvfg\" (UID: \"c2e63f29-7465-4a8d-a96f-0deb3c76b3a3\") " pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" Oct 30 16:29:55.665706 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.665671 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-xzvfg"] Oct 30 16:29:55.665976 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:29:55.665952 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-2g465 tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" podUID="c2e63f29-7465-4a8d-a96f-0deb3c76b3a3" Oct 30 16:29:55.702487 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.702449 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5875d44cbc-f9jmv"] Oct 30 16:29:55.706184 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.706162 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5875d44cbc-f9jmv" Oct 30 16:29:55.728906 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.728845 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5875d44cbc-f9jmv"] Oct 30 16:29:55.729012 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.728996 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-tls-cert\") pod \"authorino-6889cbd8c4-xzvfg\" (UID: \"c2e63f29-7465-4a8d-a96f-0deb3c76b3a3\") " pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" Oct 30 16:29:55.729077 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.729063 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g465\" (UniqueName: \"kubernetes.io/projected/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-kube-api-access-2g465\") pod \"authorino-6889cbd8c4-xzvfg\" (UID: \"c2e63f29-7465-4a8d-a96f-0deb3c76b3a3\") " pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" Oct 30 16:29:55.732072 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.732050 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-tls-cert\") pod \"authorino-6889cbd8c4-xzvfg\" (UID: \"c2e63f29-7465-4a8d-a96f-0deb3c76b3a3\") " pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" Oct 30 16:29:55.756460 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.756433 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g465\" (UniqueName: \"kubernetes.io/projected/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-kube-api-access-2g465\") pod \"authorino-6889cbd8c4-xzvfg\" (UID: \"c2e63f29-7465-4a8d-a96f-0deb3c76b3a3\") " pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" Oct 30 16:29:55.830657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.830589 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4x29\" (UniqueName: \"kubernetes.io/projected/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-kube-api-access-s4x29\") pod \"authorino-5875d44cbc-f9jmv\" (UID: \"b05fc777-cfd5-417b-b3ef-0faa3ff5238c\") " pod="kuadrant-system/authorino-5875d44cbc-f9jmv" Oct 30 16:29:55.830856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.830734 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-tls-cert\") pod \"authorino-5875d44cbc-f9jmv\" (UID: \"b05fc777-cfd5-417b-b3ef-0faa3ff5238c\") " pod="kuadrant-system/authorino-5875d44cbc-f9jmv" Oct 30 16:29:55.932135 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.932091 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-tls-cert\") pod \"authorino-5875d44cbc-f9jmv\" (UID: \"b05fc777-cfd5-417b-b3ef-0faa3ff5238c\") " pod="kuadrant-system/authorino-5875d44cbc-f9jmv" Oct 30 16:29:55.932328 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.932244 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4x29\" (UniqueName: \"kubernetes.io/projected/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-kube-api-access-s4x29\") pod \"authorino-5875d44cbc-f9jmv\" (UID: \"b05fc777-cfd5-417b-b3ef-0faa3ff5238c\") " pod="kuadrant-system/authorino-5875d44cbc-f9jmv" Oct 30 16:29:55.934620 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.934583 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-tls-cert\") pod \"authorino-5875d44cbc-f9jmv\" (UID: \"b05fc777-cfd5-417b-b3ef-0faa3ff5238c\") " pod="kuadrant-system/authorino-5875d44cbc-f9jmv" Oct 30 16:29:55.955509 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:55.955483 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4x29\" (UniqueName: \"kubernetes.io/projected/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-kube-api-access-s4x29\") pod \"authorino-5875d44cbc-f9jmv\" (UID: \"b05fc777-cfd5-417b-b3ef-0faa3ff5238c\") " pod="kuadrant-system/authorino-5875d44cbc-f9jmv" Oct 30 16:29:56.018623 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.018556 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5875d44cbc-f9jmv" Oct 30 16:29:56.312695 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.312663 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5875d44cbc-f9jmv"] Oct 30 16:29:56.315178 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:29:56.315139 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb05fc777_cfd5_417b_b3ef_0faa3ff5238c.slice/crio-7d93d4a40ce1af9ea2b06a426afa7f6dbdba8b660bccbc860c8b98d809051d36 WatchSource:0}: Error finding container 7d93d4a40ce1af9ea2b06a426afa7f6dbdba8b660bccbc860c8b98d809051d36: Status 404 returned error can't find the container with id 7d93d4a40ce1af9ea2b06a426afa7f6dbdba8b660bccbc860c8b98d809051d36 Oct 30 16:29:56.429017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.428974 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5875d44cbc-f9jmv" event={"ID":"b05fc777-cfd5-417b-b3ef-0faa3ff5238c","Type":"ContainerStarted","Data":"7d93d4a40ce1af9ea2b06a426afa7f6dbdba8b660bccbc860c8b98d809051d36"} Oct 30 16:29:56.429017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.429015 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" Oct 30 16:29:56.437586 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.437556 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" Oct 30 16:29:56.537356 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.537300 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-kube-api-access-2g465\" (UniqueName: \"kubernetes.io/projected/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-kube-api-access-2g465\") pod \"c2e63f29-7465-4a8d-a96f-0deb3c76b3a3\" (UID: \"c2e63f29-7465-4a8d-a96f-0deb3c76b3a3\") " Oct 30 16:29:56.537848 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.537417 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-tls-cert\" (UniqueName: \"kubernetes.io/secret/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-tls-cert\") pod \"c2e63f29-7465-4a8d-a96f-0deb3c76b3a3\" (UID: \"c2e63f29-7465-4a8d-a96f-0deb3c76b3a3\") " Oct 30 16:29:56.539663 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.539632 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-kube-api-access-2g465" pod "c2e63f29-7465-4a8d-a96f-0deb3c76b3a3" (UID: "c2e63f29-7465-4a8d-a96f-0deb3c76b3a3"). InnerVolumeSpecName "kube-api-access-2g465". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:29:56.539810 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.539661 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-tls-cert" pod "c2e63f29-7465-4a8d-a96f-0deb3c76b3a3" (UID: "c2e63f29-7465-4a8d-a96f-0deb3c76b3a3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:29:56.639282 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.639186 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2g465\" (UniqueName: \"kubernetes.io/projected/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-kube-api-access-2g465\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:29:56.639282 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:56.639227 6400 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3-tls-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:29:57.435519 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:57.435485 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-xzvfg" Oct 30 16:29:57.435745 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:57.435528 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5875d44cbc-f9jmv" event={"ID":"b05fc777-cfd5-417b-b3ef-0faa3ff5238c","Type":"ContainerStarted","Data":"a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d"} Oct 30 16:29:57.471838 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:57.471781 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-5875d44cbc-f9jmv" podStartSLOduration=2.023166232 podStartE2EDuration="2.471765117s" totalImagesPullingTime="448.598885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:29:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:29:57.470351281 +0000 UTC m=+506.620632268" watchObservedRunningTime="2026-10-30 16:29:57.471765117 +0000 UTC m=+506.622046109" Oct 30 16:29:57.542011 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:57.541982 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-xzvfg"] Oct 30 16:29:57.550933 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:57.550906 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-xzvfg"] Oct 30 16:29:58.966752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:29:58.966720 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2e63f29-7465-4a8d-a96f-0deb3c76b3a3" path="/var/lib/kubelet/pods/c2e63f29-7465-4a8d-a96f-0deb3c76b3a3/volumes" Oct 30 16:30:24.597287 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:30:24.597252 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:30:54.602748 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:30:54.597464 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:31:07.701801 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:07.701766 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vt287"] Oct 30 16:31:07.705426 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:07.705407 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" Oct 30 16:31:07.727114 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:07.727085 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vt287"] Oct 30 16:31:07.815128 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:07.815097 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2802841b-377f-409c-ba62-eb90a400265a-cert\") pod \"llmisvc-controller-manager-6466c554d8-vt287\" (UID: \"2802841b-377f-409c-ba62-eb90a400265a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" Oct 30 16:31:07.815320 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:07.815148 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zb7nc\" (UniqueName: \"kubernetes.io/projected/2802841b-377f-409c-ba62-eb90a400265a-kube-api-access-zb7nc\") pod \"llmisvc-controller-manager-6466c554d8-vt287\" (UID: \"2802841b-377f-409c-ba62-eb90a400265a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" Oct 30 16:31:07.916741 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:07.916706 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zb7nc\" (UniqueName: \"kubernetes.io/projected/2802841b-377f-409c-ba62-eb90a400265a-kube-api-access-zb7nc\") pod \"llmisvc-controller-manager-6466c554d8-vt287\" (UID: \"2802841b-377f-409c-ba62-eb90a400265a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" Oct 30 16:31:07.916931 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:07.916864 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2802841b-377f-409c-ba62-eb90a400265a-cert\") pod \"llmisvc-controller-manager-6466c554d8-vt287\" (UID: \"2802841b-377f-409c-ba62-eb90a400265a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" Oct 30 16:31:07.919543 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:07.919523 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2802841b-377f-409c-ba62-eb90a400265a-cert\") pod \"llmisvc-controller-manager-6466c554d8-vt287\" (UID: \"2802841b-377f-409c-ba62-eb90a400265a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" Oct 30 16:31:07.943329 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:07.938397 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zb7nc\" (UniqueName: \"kubernetes.io/projected/2802841b-377f-409c-ba62-eb90a400265a-kube-api-access-zb7nc\") pod \"llmisvc-controller-manager-6466c554d8-vt287\" (UID: \"2802841b-377f-409c-ba62-eb90a400265a\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" Oct 30 16:31:08.022395 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.022357 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" Oct 30 16:31:08.322204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.322181 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-vt287"] Oct 30 16:31:08.324767 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:31:08.324742 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2802841b_377f_409c_ba62_eb90a400265a.slice/crio-06c2f4a15293dec5d2781a66630c3bffd7793d9a38dfc646de14463769934ed4 WatchSource:0}: Error finding container 06c2f4a15293dec5d2781a66630c3bffd7793d9a38dfc646de14463769934ed4: Status 404 returned error can't find the container with id 06c2f4a15293dec5d2781a66630c3bffd7793d9a38dfc646de14463769934ed4 Oct 30 16:31:08.798484 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.798448 6400 generic.go:356] "Generic (PLEG): container finished" podID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerID="46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277" exitCode=0 Oct 30 16:31:08.799005 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.798527 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" event={"ID":"0349924c-b2b5-4a40-b6aa-9206c8046bc8","Type":"ContainerDied","Data":"46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277"} Oct 30 16:31:08.799075 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.799020 6400 scope.go:122] "RemoveContainer" containerID="46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277" Oct 30 16:31:08.800249 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.800215 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:31:08.800358 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.800304 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:31:08.800424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.800355 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:31:08.800424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.800227 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" event={"ID":"2802841b-377f-409c-ba62-eb90a400265a","Type":"ContainerStarted","Data":"63cde4379e3b422074fb943514903df1eaaf2f006b016b0279e72ebff6acbba9"} Oct 30 16:31:08.800525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.800435 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" event={"ID":"2802841b-377f-409c-ba62-eb90a400265a","Type":"ContainerStarted","Data":"06c2f4a15293dec5d2781a66630c3bffd7793d9a38dfc646de14463769934ed4"} Oct 30 16:31:08.800525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.800484 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" Oct 30 16:31:08.893463 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:08.892852 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" podStartSLOduration=1.545873432 podStartE2EDuration="1.89282328s" totalImagesPullingTime="346.949848ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:31:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:31:08.890417285 +0000 UTC m=+578.040698275" watchObservedRunningTime="2026-10-30 16:31:08.89282328 +0000 UTC m=+578.043104273" Oct 30 16:31:09.808104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:09.808070 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" event={"ID":"0349924c-b2b5-4a40-b6aa-9206c8046bc8","Type":"ContainerStarted","Data":"3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5"} Oct 30 16:31:09.832628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:09.832001 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:31:09.832628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:09.832143 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xpkrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 16:31:09.832628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:09.832196 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 16:31:10.834668 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:10.831701 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xpkrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 16:31:10.834668 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:10.831770 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 16:31:11.830628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:11.830018 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xpkrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 16:31:11.830628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:11.830086 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 16:31:12.814631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:12.811024 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xpkrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 16:31:12.814631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:12.811090 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 16:31:13.814637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:13.814016 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xpkrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 16:31:13.814637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:13.814080 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 16:31:14.811636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:14.811581 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-xpkrw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body= Oct 30 16:31:14.811874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:14.811662 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" Oct 30 16:31:15.815621 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:15.815573 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:31:15.816113 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:15.816101 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:31:15.816823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:15.816800 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:31:24.596867 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:31:24.596798 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:31:39.809939 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:39.809908 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-vt287" Oct 30 16:31:39.873996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:39.873963 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv"] Oct 30 16:31:39.874212 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:39.874190 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" podUID="ed0131d6-c205-4bd0-9169-63656361235d" containerName="manager" containerID="cri-o://d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6" gracePeriod=30 Oct 30 16:31:40.123310 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.123281 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:31:40.221252 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.221212 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed0131d6-c205-4bd0-9169-63656361235d-cert\" (UniqueName: \"kubernetes.io/secret/ed0131d6-c205-4bd0-9169-63656361235d-cert\") pod \"ed0131d6-c205-4bd0-9169-63656361235d\" (UID: \"ed0131d6-c205-4bd0-9169-63656361235d\") " Oct 30 16:31:40.221425 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.221315 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed0131d6-c205-4bd0-9169-63656361235d-kube-api-access-pbfdk\" (UniqueName: \"kubernetes.io/projected/ed0131d6-c205-4bd0-9169-63656361235d-kube-api-access-pbfdk\") pod \"ed0131d6-c205-4bd0-9169-63656361235d\" (UID: \"ed0131d6-c205-4bd0-9169-63656361235d\") " Oct 30 16:31:40.223594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.223563 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed0131d6-c205-4bd0-9169-63656361235d-cert" pod "ed0131d6-c205-4bd0-9169-63656361235d" (UID: "ed0131d6-c205-4bd0-9169-63656361235d"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:31:40.223742 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.223717 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed0131d6-c205-4bd0-9169-63656361235d-kube-api-access-pbfdk" pod "ed0131d6-c205-4bd0-9169-63656361235d" (UID: "ed0131d6-c205-4bd0-9169-63656361235d"). InnerVolumeSpecName "kube-api-access-pbfdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:31:40.322469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.322434 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbfdk\" (UniqueName: \"kubernetes.io/projected/ed0131d6-c205-4bd0-9169-63656361235d-kube-api-access-pbfdk\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:31:40.322469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.322464 6400 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ed0131d6-c205-4bd0-9169-63656361235d-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:31:40.986720 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.986688 6400 generic.go:356] "Generic (PLEG): container finished" podID="ed0131d6-c205-4bd0-9169-63656361235d" containerID="d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6" exitCode=0 Oct 30 16:31:40.987098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.986772 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" event={"ID":"ed0131d6-c205-4bd0-9169-63656361235d","Type":"ContainerDied","Data":"d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6"} Oct 30 16:31:40.987098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.986802 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" Oct 30 16:31:40.987098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.986828 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv" event={"ID":"ed0131d6-c205-4bd0-9169-63656361235d","Type":"ContainerDied","Data":"5861694ec602b0bb6e53b8272523edc69a10e4e3ebc46540f8f02175b2cfa404"} Oct 30 16:31:40.987098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.986858 6400 scope.go:122] "RemoveContainer" containerID="d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6" Oct 30 16:31:40.995675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.995654 6400 scope.go:122] "RemoveContainer" containerID="d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6" Oct 30 16:31:40.995947 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:31:40.995921 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6\": container with ID starting with d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6 not found: ID does not exist" containerID="d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6" Oct 30 16:31:40.995993 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:40.995958 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6"} err="failed to get container status \"d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6\": rpc error: code = NotFound desc = could not find container \"d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6\": container with ID starting with d105e0ae55ad407751b01852121681c321ff2a47d3567f3f72e1b9b94338d4f6 not found: ID does not exist" Oct 30 16:31:41.030892 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:41.030867 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv"] Oct 30 16:31:41.034345 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:41.034326 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-nxcfv"] Oct 30 16:31:42.965474 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:31:42.965435 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed0131d6-c205-4bd0-9169-63656361235d" path="/var/lib/kubelet/pods/ed0131d6-c205-4bd0-9169-63656361235d/volumes" Oct 30 16:31:54.597470 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:31:54.597435 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:32:24.597114 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:32:24.597077 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:32:28.925843 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:28.925800 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5875d44cbc-f9jmv"] Oct 30 16:32:28.926501 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:28.926438 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-5875d44cbc-f9jmv" podUID="b05fc777-cfd5-417b-b3ef-0faa3ff5238c" containerName="authorino" containerID="cri-o://a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d" gracePeriod=30 Oct 30 16:32:29.176984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.176919 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5875d44cbc-f9jmv" Oct 30 16:32:29.196343 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.196309 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-kube-api-access-s4x29\" (UniqueName: \"kubernetes.io/projected/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-kube-api-access-s4x29\") pod \"b05fc777-cfd5-417b-b3ef-0faa3ff5238c\" (UID: \"b05fc777-cfd5-417b-b3ef-0faa3ff5238c\") " Oct 30 16:32:29.196496 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.196402 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-tls-cert\" (UniqueName: \"kubernetes.io/secret/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-tls-cert\") pod \"b05fc777-cfd5-417b-b3ef-0faa3ff5238c\" (UID: \"b05fc777-cfd5-417b-b3ef-0faa3ff5238c\") " Oct 30 16:32:29.198658 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.198631 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-kube-api-access-s4x29" pod "b05fc777-cfd5-417b-b3ef-0faa3ff5238c" (UID: "b05fc777-cfd5-417b-b3ef-0faa3ff5238c"). InnerVolumeSpecName "kube-api-access-s4x29". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:32:29.207854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.207831 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-tls-cert" pod "b05fc777-cfd5-417b-b3ef-0faa3ff5238c" (UID: "b05fc777-cfd5-417b-b3ef-0faa3ff5238c"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:32:29.248650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.248581 6400 generic.go:356] "Generic (PLEG): container finished" podID="b05fc777-cfd5-417b-b3ef-0faa3ff5238c" containerID="a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d" exitCode=0 Oct 30 16:32:29.248849 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.248657 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5875d44cbc-f9jmv" Oct 30 16:32:29.248849 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.248652 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5875d44cbc-f9jmv" event={"ID":"b05fc777-cfd5-417b-b3ef-0faa3ff5238c","Type":"ContainerDied","Data":"a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d"} Oct 30 16:32:29.248849 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.248768 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5875d44cbc-f9jmv" event={"ID":"b05fc777-cfd5-417b-b3ef-0faa3ff5238c","Type":"ContainerDied","Data":"7d93d4a40ce1af9ea2b06a426afa7f6dbdba8b660bccbc860c8b98d809051d36"} Oct 30 16:32:29.248849 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.248792 6400 scope.go:122] "RemoveContainer" containerID="a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d" Oct 30 16:32:29.258687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.258668 6400 scope.go:122] "RemoveContainer" containerID="a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d" Oct 30 16:32:29.258953 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:32:29.258936 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d\": container with ID starting with a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d not found: ID does not exist" containerID="a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d" Oct 30 16:32:29.259003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.258965 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d"} err="failed to get container status \"a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d\": rpc error: code = NotFound desc = could not find container \"a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d\": container with ID starting with a9f902029a8a6e31417c0918f17e27ff8d0237af4f9e620b3bc7288b49e3927d not found: ID does not exist" Oct 30 16:32:29.297285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.297248 6400 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-tls-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:32:29.297285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.297279 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s4x29\" (UniqueName: \"kubernetes.io/projected/b05fc777-cfd5-417b-b3ef-0faa3ff5238c-kube-api-access-s4x29\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:32:29.298982 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.298956 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5875d44cbc-f9jmv"] Oct 30 16:32:29.300941 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:29.300914 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5875d44cbc-f9jmv"] Oct 30 16:32:30.966221 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:30.966193 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b05fc777-cfd5-417b-b3ef-0faa3ff5238c" path="/var/lib/kubelet/pods/b05fc777-cfd5-417b-b3ef-0faa3ff5238c/volumes" Oct 30 16:32:46.730186 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.730153 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj"] Oct 30 16:32:46.730784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.730682 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ed0131d6-c205-4bd0-9169-63656361235d" containerName="manager" Oct 30 16:32:46.730784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.730701 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed0131d6-c205-4bd0-9169-63656361235d" containerName="manager" Oct 30 16:32:46.730784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.730720 6400 cpu_manager.go:414] "containerMap: removing container" podUID="b05fc777-cfd5-417b-b3ef-0faa3ff5238c" containerName="authorino" Oct 30 16:32:46.730784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.730728 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b05fc777-cfd5-417b-b3ef-0faa3ff5238c" containerName="authorino" Oct 30 16:32:46.730915 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.730796 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed0131d6-c205-4bd0-9169-63656361235d" containerName="manager" Oct 30 16:32:46.730915 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.730806 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="b05fc777-cfd5-417b-b3ef-0faa3ff5238c" containerName="authorino" Oct 30 16:32:46.732910 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.732895 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.761464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.761427 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj"] Oct 30 16:32:46.857482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.857445 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.857482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.857481 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.857746 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.857501 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.857746 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.857567 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.857746 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.857699 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.857746 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.857726 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.857746 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.857743 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.857920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.857761 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwsp5\" (UniqueName: \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-kube-api-access-kwsp5\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.857953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.857916 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959128 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959096 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959153 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959182 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959208 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959237 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959268 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959291 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959312 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959742 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959339 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwsp5\" (UniqueName: \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-kube-api-access-kwsp5\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959742 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959732 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.959856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.959790 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.960096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.960073 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.960204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.960110 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.960204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.960140 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.961832 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.961808 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.961949 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.961909 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.980690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.980629 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:46.980690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:46.980682 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwsp5\" (UniqueName: \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-kube-api-access-kwsp5\") pod \"maas-default-gateway-openshift-default-76f4876667-srnkj\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:47.046337 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:47.046304 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:47.359690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:47.359666 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj"] Oct 30 16:32:47.361932 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:32:47.361906 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4d27ba0_2467_4de7_bf62_a064e2279d1f.slice/crio-305e84ea6971a0ed3d398d992730ddfbf059898b4c3abb85084737b5d8604cf3 WatchSource:0}: Error finding container 305e84ea6971a0ed3d398d992730ddfbf059898b4c3abb85084737b5d8604cf3: Status 404 returned error can't find the container with id 305e84ea6971a0ed3d398d992730ddfbf059898b4c3abb85084737b5d8604cf3 Oct 30 16:32:47.363846 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:47.363815 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:32:47.363935 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:47.363922 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:32:47.363985 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:47.363973 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:32:48.345096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:48.345057 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" event={"ID":"c4d27ba0-2467-4de7-bf62-a064e2279d1f","Type":"ContainerStarted","Data":"83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa"} Oct 30 16:32:48.345526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:48.345119 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" event={"ID":"c4d27ba0-2467-4de7-bf62-a064e2279d1f","Type":"ContainerStarted","Data":"305e84ea6971a0ed3d398d992730ddfbf059898b4c3abb85084737b5d8604cf3"} Oct 30 16:32:48.404902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:48.404851 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" podStartSLOduration=2.4048374790000002 podStartE2EDuration="2.404837479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:32:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:32:48.3944425 +0000 UTC m=+677.544723511" watchObservedRunningTime="2026-10-30 16:32:48.404837479 +0000 UTC m=+677.555118446" Oct 30 16:32:49.047281 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:49.047239 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:49.048709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:49.048683 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-srnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:32:49.048846 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:49.048744 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 16:32:50.047363 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:50.047330 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-srnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:32:50.047826 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:50.047412 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 16:32:51.047037 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:51.047003 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-srnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:32:51.047198 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:51.047058 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 16:32:52.047207 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:52.047165 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-srnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:32:52.047677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:52.047236 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 16:32:53.046805 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:53.046769 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-srnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:32:53.046986 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:53.046837 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 16:32:54.047582 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:54.047538 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-srnkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" start-of-body= Oct 30 16:32:54.048096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:54.047637 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.47:15021/healthz/ready\": dial tcp 10.134.0.47:15021: connect: connection refused" Oct 30 16:32:54.601211 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:32:54.601163 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:32:55.552230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:55.552194 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:55.552747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:55.552632 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:32:55.553354 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:32:55.553333 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:33:24.597380 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:33:24.597339 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:33:54.597937 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:33:54.597896 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:34:12.993034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:12.993002 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v"] Oct 30 16:34:12.997335 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:12.997301 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.010491 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.010461 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-a8c22d-openshift-default-dockercfg-qfhzt\"" Oct 30 16:34:13.027717 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.027683 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v"] Oct 30 16:34:13.061117 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.061063 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-certs\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.061315 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.061139 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-credential-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.061315 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.061204 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxfn\" (UniqueName: \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-kube-api-access-skxfn\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.061315 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.061265 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-envoy\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.061315 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.061291 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608709cd-a150-41dd-ad88-ea0f0040de82-istio-podinfo\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.061518 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.061408 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608709cd-a150-41dd-ad88-ea0f0040de82-istiod-ca-cert\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.061518 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.061449 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-istio-token\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.061766 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.061740 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.061872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.061788 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-data\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.162797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.162759 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-credential-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.162797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.162802 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxfn\" (UniqueName: \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-kube-api-access-skxfn\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.163071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.162820 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-envoy\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.163071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.162837 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608709cd-a150-41dd-ad88-ea0f0040de82-istio-podinfo\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.163071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.162892 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608709cd-a150-41dd-ad88-ea0f0040de82-istiod-ca-cert\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.163071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.162923 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-istio-token\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.163071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.162962 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.163071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.162992 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-data\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.163071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.163035 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-certs\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.163491 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.163464 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-certs\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.163799 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.163768 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-credential-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.165692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.165088 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-data\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.165692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.165175 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608709cd-a150-41dd-ad88-ea0f0040de82-istiod-ca-cert\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.166230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.166201 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-socket\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.169643 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.169596 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608709cd-a150-41dd-ad88-ea0f0040de82-istio-podinfo\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.174845 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.174822 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-envoy\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.185918 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.185859 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxfn\" (UniqueName: \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-kube-api-access-skxfn\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.186469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.186446 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-istio-token\") pod \"e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.312970 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.312865 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:13.630082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.630048 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v"] Oct 30 16:34:13.631853 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:34:13.631815 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608709cd_a150_41dd_ad88_ea0f0040de82.slice/crio-af8bd9ffa09fe36302428df54643a8fdd6fcac72b26648922f1a26bb3ff4cfda WatchSource:0}: Error finding container af8bd9ffa09fe36302428df54643a8fdd6fcac72b26648922f1a26bb3ff4cfda: Status 404 returned error can't find the container with id af8bd9ffa09fe36302428df54643a8fdd6fcac72b26648922f1a26bb3ff4cfda Oct 30 16:34:13.634183 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.634149 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:13.634264 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.634212 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:13.634264 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.634241 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:13.778514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.778463 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" event={"ID":"608709cd-a150-41dd-ad88-ea0f0040de82","Type":"ContainerStarted","Data":"82b95fd9ba5127519f332fd844fd32932442a789bec5053913cd873a1a2e32ac"} Oct 30 16:34:13.778719 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.778531 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" event={"ID":"608709cd-a150-41dd-ad88-ea0f0040de82","Type":"ContainerStarted","Data":"af8bd9ffa09fe36302428df54643a8fdd6fcac72b26648922f1a26bb3ff4cfda"} Oct 30 16:34:13.822284 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:13.822204 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" podStartSLOduration=1.8221897 podStartE2EDuration="1.8221897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:13.822070093 +0000 UTC m=+762.972351094" watchObservedRunningTime="2026-10-30 16:34:13.8221897 +0000 UTC m=+762.972470688" Oct 30 16:34:14.313935 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:14.313896 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:14.318933 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:14.318907 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:14.783237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:14.783194 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:14.784143 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:14.784121 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:24.252693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.252656 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz"] Oct 30 16:34:24.256315 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.256298 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.277750 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.277723 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-d693d0-serving-cert\"" Oct 30 16:34:24.279273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.279242 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5jjc6\"" Oct 30 16:34:24.279273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.279242 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 16:34:24.282136 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.282110 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz"] Oct 30 16:34:24.373742 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.373707 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.373742 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.373746 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.373983 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.373768 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89gh4\" (UniqueName: \"kubernetes.io/projected/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-kube-api-access-89gh4\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.373983 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.373900 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-metrics-certs\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.475080 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.475038 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-metrics-certs\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.475294 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.475158 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.475294 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.475190 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.475294 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.475219 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89gh4\" (UniqueName: \"kubernetes.io/projected/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-kube-api-access-89gh4\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.475939 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.475864 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.477738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.477710 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-metrics-certs\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.477834 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.477735 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-maas-api-tls\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.497403 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.497368 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89gh4\" (UniqueName: \"kubernetes.io/projected/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-kube-api-access-89gh4\") pod \"maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.554392 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.554300 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v"] Oct 30 16:34:24.560095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.560040 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.572972 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.572721 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:24.576007 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.575984 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"simulator-w4-263f00-kserve-self-signed-certs\"" Oct 30 16:34:24.576279 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.576214 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"kube-root-ca.crt\"" Oct 30 16:34:24.576355 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.576290 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"default-dockercfg-b54g7\"" Oct 30 16:34:24.576412 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.576363 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"openshift-service-ca.crt\"" Oct 30 16:34:24.601424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.598913 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v"] Oct 30 16:34:24.608268 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:34:24.608226 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:34:24.676330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.676291 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-dshm\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.676505 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.676340 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-model-cache\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.676505 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.676367 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbc7l\" (UniqueName: \"kubernetes.io/projected/f0c5e6d3-f198-4daa-891f-6af1994e5d89-kube-api-access-dbc7l\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.676586 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.676521 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-home\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.676640 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.676625 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tmp-dir\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.676715 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.676700 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tls-certs\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.778037 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.778001 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-dshm\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.778037 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.778041 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-model-cache\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.778293 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.778059 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbc7l\" (UniqueName: \"kubernetes.io/projected/f0c5e6d3-f198-4daa-891f-6af1994e5d89-kube-api-access-dbc7l\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.778293 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.778093 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-home\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.778293 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.778146 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tmp-dir\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.778293 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.778194 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tls-certs\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.778548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.778523 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-model-cache\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.778548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.778533 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tmp-dir\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.778708 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.778587 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-home\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.780573 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.780553 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-dshm\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.780855 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.780839 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tls-certs\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.800138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.800108 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbc7l\" (UniqueName: \"kubernetes.io/projected/f0c5e6d3-f198-4daa-891f-6af1994e5d89-kube-api-access-dbc7l\") pod \"simulator-w4-263f00-kserve-777999d56d-lkk8v\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.873984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.873887 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:24.891920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:24.891890 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz"] Oct 30 16:34:24.893868 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:34:24.893824 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ae4c919_e0af_4d39_98ef_5fb6eceab7d0.slice/crio-af14a15ed8d699a329c9025624c40a553a988623230e691be5dcc79a91caeda9 WatchSource:0}: Error finding container af14a15ed8d699a329c9025624c40a553a988623230e691be5dcc79a91caeda9: Status 404 returned error can't find the container with id af14a15ed8d699a329c9025624c40a553a988623230e691be5dcc79a91caeda9 Oct 30 16:34:25.184710 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:25.183858 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v"] Oct 30 16:34:25.188862 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:34:25.188823 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0c5e6d3_f198_4daa_891f_6af1994e5d89.slice/crio-35605c825024a5f407156a7899c3c069b1f8bb77b4f2cd0bc96dbc3b6b626e01 WatchSource:0}: Error finding container 35605c825024a5f407156a7899c3c069b1f8bb77b4f2cd0bc96dbc3b6b626e01: Status 404 returned error can't find the container with id 35605c825024a5f407156a7899c3c069b1f8bb77b4f2cd0bc96dbc3b6b626e01 Oct 30 16:34:25.842633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:25.842563 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" event={"ID":"f0c5e6d3-f198-4daa-891f-6af1994e5d89","Type":"ContainerStarted","Data":"35605c825024a5f407156a7899c3c069b1f8bb77b4f2cd0bc96dbc3b6b626e01"} Oct 30 16:34:25.845157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:25.845130 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" event={"ID":"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0","Type":"ContainerStarted","Data":"af14a15ed8d699a329c9025624c40a553a988623230e691be5dcc79a91caeda9"} Oct 30 16:34:27.858375 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:27.858346 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" event={"ID":"f0c5e6d3-f198-4daa-891f-6af1994e5d89","Type":"ContainerStarted","Data":"6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278"} Oct 30 16:34:27.874043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:27.874006 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:27.897665 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:27.897069 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podStartSLOduration=2.304598635 podStartE2EDuration="3.897048847s" totalImagesPullingTime="1.592450212s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:27.895758011 +0000 UTC m=+777.046038998" watchObservedRunningTime="2026-10-30 16:34:27.897048847 +0000 UTC m=+777.047329832" Oct 30 16:34:28.865907 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:28.865872 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" event={"ID":"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0","Type":"ContainerStarted","Data":"fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060"} Oct 30 16:34:28.866330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:28.865992 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:28.904393 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:28.904324 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" podStartSLOduration=1.999694364 podStartE2EDuration="4.904305741s" totalImagesPullingTime="2.904611377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:28.902801608 +0000 UTC m=+778.053082596" watchObservedRunningTime="2026-10-30 16:34:28.904305741 +0000 UTC m=+778.054586731" Oct 30 16:34:34.874916 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:34.874878 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:34.876778 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:34.876748 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 16:34:34.887643 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:34.887625 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:34.905542 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:34.905516 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:34:36.444012 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.443980 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z"] Oct 30 16:34:36.447800 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.447777 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.468457 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.468429 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0b2348\"/\"openshift-service-ca.crt\"" Oct 30 16:34:36.478456 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.478428 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-0b2348\"/\"kube-root-ca.crt\"" Oct 30 16:34:36.478714 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.478435 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0b2348\"/\"default-dockercfg-j8sz5\"" Oct 30 16:34:36.478845 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.478435 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-0b2348\"/\"simulator-w3-0b2348-kserve-self-signed-certs\"" Oct 30 16:34:36.481382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.481359 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z"] Oct 30 16:34:36.607791 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.607741 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-home\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.607791 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.607801 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-model-cache\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.608052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.607943 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tls-certs\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.608052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.607984 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tmp-dir\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.608052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.608004 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k46sh\" (UniqueName: \"kubernetes.io/projected/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-kube-api-access-k46sh\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.608052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.608044 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-dshm\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.709152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.709075 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-model-cache\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.709152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.709118 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tls-certs\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.709152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.709153 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tmp-dir\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.709436 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.709168 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k46sh\" (UniqueName: \"kubernetes.io/projected/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-kube-api-access-k46sh\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.709436 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.709184 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-dshm\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.709436 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.709252 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-home\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.709592 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.709548 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-model-cache\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.709674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.709592 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tmp-dir\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.709674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.709649 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-home\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.711771 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.711746 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-dshm\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.711973 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.711955 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tls-certs\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.730901 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.730878 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k46sh\" (UniqueName: \"kubernetes.io/projected/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-kube-api-access-k46sh\") pod \"simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:36.762989 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:36.762955 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:37.067217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:37.067189 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z"] Oct 30 16:34:37.069732 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:34:37.069701 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c1fd2b9_f051_4cbe_a305_57d6136e2cf7.slice/crio-7c4d7bb735df152c51ef91a4fea7995ff5f6a5f8b5e5287abf36233f09e0b19e WatchSource:0}: Error finding container 7c4d7bb735df152c51ef91a4fea7995ff5f6a5f8b5e5287abf36233f09e0b19e: Status 404 returned error can't find the container with id 7c4d7bb735df152c51ef91a4fea7995ff5f6a5f8b5e5287abf36233f09e0b19e Oct 30 16:34:37.924194 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:37.924146 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" event={"ID":"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7","Type":"ContainerStarted","Data":"046785d25e24219b5687975012a2edd9135cc8cf93d378583f266acf4c1079b7"} Oct 30 16:34:37.924752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:37.924206 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" event={"ID":"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7","Type":"ContainerStarted","Data":"7c4d7bb735df152c51ef91a4fea7995ff5f6a5f8b5e5287abf36233f09e0b19e"} Oct 30 16:34:37.964522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:37.964465 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podStartSLOduration=1.964451383 podStartE2EDuration="1.964451383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:37.964242701 +0000 UTC m=+787.114523702" watchObservedRunningTime="2026-10-30 16:34:37.964451383 +0000 UTC m=+787.114732371" Oct 30 16:34:38.763737 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:38.763686 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:44.475648 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.475595 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8"] Oct 30 16:34:44.479547 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.479495 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.485695 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.485665 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-home\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.486016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.485805 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tmp-dir\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.486016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.485844 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-model-cache\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.486016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.485873 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-dshm\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.486016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.485923 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltmgp\" (UniqueName: \"kubernetes.io/projected/c890bd5e-2813-4f86-ab91-61f5334d2f2b-kube-api-access-ltmgp\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.486016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.485981 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tls-certs\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.494735 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.494713 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-750dcb\"/\"kube-root-ca.crt\"" Oct 30 16:34:44.494735 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.494715 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-750dcb\"/\"openshift-service-ca.crt\"" Oct 30 16:34:44.504284 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.504259 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-750dcb\"/\"simulator-w2-750dcb-kserve-self-signed-certs\"" Oct 30 16:34:44.505100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.504340 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-750dcb\"/\"default-dockercfg-wnttz\"" Oct 30 16:34:44.508382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.508359 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8"] Oct 30 16:34:44.586469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.586433 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltmgp\" (UniqueName: \"kubernetes.io/projected/c890bd5e-2813-4f86-ab91-61f5334d2f2b-kube-api-access-ltmgp\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.586709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.586478 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tls-certs\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.586709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.586531 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-home\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.586709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.586555 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tmp-dir\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.586709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.586573 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-model-cache\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.586931 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.586720 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-dshm\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.587004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.586985 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tmp-dir\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.587335 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.587310 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-home\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.587478 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.587340 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-model-cache\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.589311 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.589283 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-dshm\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.589840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.589805 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tls-certs\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.612818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.612788 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltmgp\" (UniqueName: \"kubernetes.io/projected/c890bd5e-2813-4f86-ab91-61f5334d2f2b-kube-api-access-ltmgp\") pod \"simulator-w2-750dcb-kserve-8649fb579d-m4wn8\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:44.794987 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:44.794940 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:45.101942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:45.101918 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8"] Oct 30 16:34:45.104900 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:34:45.104869 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc890bd5e_2813_4f86_ab91_61f5334d2f2b.slice/crio-d4931813c312884f5fc32010416004ac94efe9e5f990a1522d6104a0691650e5 WatchSource:0}: Error finding container d4931813c312884f5fc32010416004ac94efe9e5f990a1522d6104a0691650e5: Status 404 returned error can't find the container with id d4931813c312884f5fc32010416004ac94efe9e5f990a1522d6104a0691650e5 Oct 30 16:34:45.823774 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:45.823728 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:34:45.824159 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:45.823789 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:34:45.981916 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:45.981873 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" event={"ID":"c890bd5e-2813-4f86-ab91-61f5334d2f2b","Type":"ContainerStarted","Data":"1b795b041292e26e1bcf6a4e0642022341e43d6adcd92b58987397c774e59571"} Oct 30 16:34:45.982102 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:45.981927 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" event={"ID":"c890bd5e-2813-4f86-ab91-61f5334d2f2b","Type":"ContainerStarted","Data":"d4931813c312884f5fc32010416004ac94efe9e5f990a1522d6104a0691650e5"} Oct 30 16:34:46.021238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.021180 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podStartSLOduration=2.021165333 podStartE2EDuration="2.021165333s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:46.019193505 +0000 UTC m=+795.169474495" watchObservedRunningTime="2026-10-30 16:34:46.021165333 +0000 UTC m=+795.171446321" Oct 30 16:34:46.724234 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.724197 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj"] Oct 30 16:34:46.727115 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.727092 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.749747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.749718 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3e2f96\"/\"default-dockercfg-vbbmf\"" Oct 30 16:34:46.749747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.749734 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3e2f96\"/\"openshift-service-ca.crt\"" Oct 30 16:34:46.749948 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.749834 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-3e2f96\"/\"kube-root-ca.crt\"" Oct 30 16:34:46.752051 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.752025 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj"] Oct 30 16:34:46.763242 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.763206 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:46.781426 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.781395 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:46.795439 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.795410 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:46.811852 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.811777 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cf4325-2f94-455f-82b5-f33ad48c3884-tls-certs\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.811852 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.811822 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chmfz\" (UniqueName: \"kubernetes.io/projected/32cf4325-2f94-455f-82b5-f33ad48c3884-kube-api-access-chmfz\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.812088 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.812011 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-home\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.812088 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.812040 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-dshm\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.812224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.812199 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-model-cache\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.812278 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.812251 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-tmp-dir\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.815976 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.815956 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3e2f96\"/\"simulator-w0-3e2f96-kserve-self-signed-certs\"" Oct 30 16:34:46.913755 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.913719 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cf4325-2f94-455f-82b5-f33ad48c3884-tls-certs\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.914243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.913770 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chmfz\" (UniqueName: \"kubernetes.io/projected/32cf4325-2f94-455f-82b5-f33ad48c3884-kube-api-access-chmfz\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.914243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.913841 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-home\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.914243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.913865 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-dshm\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.914243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.913950 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-model-cache\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.914243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.913986 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-tmp-dir\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.914520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.914242 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-home\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.914520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.914336 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-tmp-dir\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.914520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.914353 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-model-cache\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.916658 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.916630 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cf4325-2f94-455f-82b5-f33ad48c3884-tls-certs\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.916795 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.916780 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-dshm\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.941651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.941565 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chmfz\" (UniqueName: \"kubernetes.io/projected/32cf4325-2f94-455f-82b5-f33ad48c3884-kube-api-access-chmfz\") pod \"simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:46.988552 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.988515 6400 generic.go:356] "Generic (PLEG): container finished" podID="608709cd-a150-41dd-ad88-ea0f0040de82" containerID="82b95fd9ba5127519f332fd844fd32932442a789bec5053913cd873a1a2e32ac" exitCode=0 Oct 30 16:34:46.988733 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.988589 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" event={"ID":"608709cd-a150-41dd-ad88-ea0f0040de82","Type":"ContainerDied","Data":"82b95fd9ba5127519f332fd844fd32932442a789bec5053913cd873a1a2e32ac"} Oct 30 16:34:46.989144 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.989123 6400 scope.go:122] "RemoveContainer" containerID="82b95fd9ba5127519f332fd844fd32932442a789bec5053913cd873a1a2e32ac" Oct 30 16:34:46.990287 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.990258 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:46.990404 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.990334 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:46.990404 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:46.990367 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:34:47.002835 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:47.001786 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:34:47.048939 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:47.048915 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:47.381820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:47.381789 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj"] Oct 30 16:34:47.385935 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:34:47.385904 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod32cf4325_2f94_455f_82b5_f33ad48c3884.slice/crio-fdc50db3f28540e9235118fe9e9dfdc9dbeadaa33d9f0ca989bd6e49cd15f994 WatchSource:0}: Error finding container fdc50db3f28540e9235118fe9e9dfdc9dbeadaa33d9f0ca989bd6e49cd15f994: Status 404 returned error can't find the container with id fdc50db3f28540e9235118fe9e9dfdc9dbeadaa33d9f0ca989bd6e49cd15f994 Oct 30 16:34:47.996046 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:47.996011 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" event={"ID":"608709cd-a150-41dd-ad88-ea0f0040de82","Type":"ContainerStarted","Data":"d46e54fdba2c9869b091e93d6c24db73c4e54ca62579e24a7c4567aa83cdaac1"} Oct 30 16:34:47.997913 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:47.997871 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" event={"ID":"32cf4325-2f94-455f-82b5-f33ad48c3884","Type":"ContainerStarted","Data":"2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843"} Oct 30 16:34:47.997913 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:47.997908 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" event={"ID":"32cf4325-2f94-455f-82b5-f33ad48c3884","Type":"ContainerStarted","Data":"fdc50db3f28540e9235118fe9e9dfdc9dbeadaa33d9f0ca989bd6e49cd15f994"} Oct 30 16:34:48.049717 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:48.049676 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:48.133627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:48.132763 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podStartSLOduration=2.132736382 podStartE2EDuration="2.132736382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:48.108963069 +0000 UTC m=+797.259244057" watchObservedRunningTime="2026-10-30 16:34:48.132736382 +0000 UTC m=+797.283017375" Oct 30 16:34:48.329686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:48.315007 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:48.329686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:48.315132 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:34:48.329686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:48.315188 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 16:34:49.314303 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:49.314263 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:34:49.315208 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:49.315168 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 16:34:50.319638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:50.319020 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:34:50.319638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:50.319091 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 16:34:51.321636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:51.314188 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:34:51.321636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:51.314247 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 16:34:52.318673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.316997 6400 patch_prober.go:29] interesting pod/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" start-of-body= Oct 30 16:34:52.318673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.317060 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.48:15021/healthz/ready\": dial tcp 10.134.0.48:15021: connect: connection refused" Oct 30 16:34:52.838851 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.838811 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz"] Oct 30 16:34:52.843210 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.843172 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.866160 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.866131 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-263f00\"/\"premium-w4-263f00-kserve-self-signed-certs\"" Oct 30 16:34:52.868759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.868725 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz"] Oct 30 16:34:52.889633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.889574 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-model-cache\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.889633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.889636 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzwjn\" (UniqueName: \"kubernetes.io/projected/19ea2eab-3e12-4193-8f63-aa7872b284c2-kube-api-access-vzwjn\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.889837 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.889749 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-dshm\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.889837 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.889789 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-tmp-dir\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.889903 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.889871 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19ea2eab-3e12-4193-8f63-aa7872b284c2-tls-certs\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.889933 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.889916 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-home\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.990495 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.990462 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-home\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.990726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.990568 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-model-cache\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.990726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.990623 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzwjn\" (UniqueName: \"kubernetes.io/projected/19ea2eab-3e12-4193-8f63-aa7872b284c2-kube-api-access-vzwjn\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.990726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.990676 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-dshm\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.990726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.990713 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-tmp-dir\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.990941 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.990775 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19ea2eab-3e12-4193-8f63-aa7872b284c2-tls-certs\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.991391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.991285 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-home\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.991526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.991505 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-model-cache\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.991594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.991556 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-tmp-dir\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.993996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.993970 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-dshm\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:52.994384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:52.994366 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19ea2eab-3e12-4193-8f63-aa7872b284c2-tls-certs\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:53.018476 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:53.018441 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzwjn\" (UniqueName: \"kubernetes.io/projected/19ea2eab-3e12-4193-8f63-aa7872b284c2-kube-api-access-vzwjn\") pod \"premium-w4-263f00-kserve-699df8c945-c4pkz\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:53.158814 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:53.158714 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:53.317345 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:53.317315 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:53.317744 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:53.317723 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:53.318407 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:53.318383 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:34:53.472396 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:53.472361 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz"] Oct 30 16:34:53.474589 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:34:53.474543 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19ea2eab_3e12_4193_8f63_aa7872b284c2.slice/crio-9469783f6d91a1e76ebea7bda6d87f1b2071fa8692ca24296fe1ef6284c1a107 WatchSource:0}: Error finding container 9469783f6d91a1e76ebea7bda6d87f1b2071fa8692ca24296fe1ef6284c1a107: Status 404 returned error can't find the container with id 9469783f6d91a1e76ebea7bda6d87f1b2071fa8692ca24296fe1ef6284c1a107 Oct 30 16:34:54.060825 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:54.060786 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" event={"ID":"19ea2eab-3e12-4193-8f63-aa7872b284c2","Type":"ContainerStarted","Data":"bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16"} Oct 30 16:34:54.061239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:54.060846 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" event={"ID":"19ea2eab-3e12-4193-8f63-aa7872b284c2","Type":"ContainerStarted","Data":"9469783f6d91a1e76ebea7bda6d87f1b2071fa8692ca24296fe1ef6284c1a107"} Oct 30 16:34:54.100194 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:54.100142 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podStartSLOduration=2.10012407 podStartE2EDuration="2.10012407s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:34:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:34:54.098184319 +0000 UTC m=+803.248465306" watchObservedRunningTime="2026-10-30 16:34:54.10012407 +0000 UTC m=+803.250405059" Oct 30 16:34:54.159693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:54.159663 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:34:54.597648 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:34:54.597596 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:34:54.795876 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:54.795841 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:54.808969 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:54.808941 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:55.076766 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:55.076739 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:34:57.049392 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:57.049353 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:57.062029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:57.062002 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:34:57.089685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:34:57.089660 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:35:01.060858 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.060819 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j"] Oct 30 16:35:01.068357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.068331 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.093801 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.093770 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-967fac\"/\"simulator-w5-967fac-kserve-self-signed-certs\"" Oct 30 16:35:01.093987 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.093814 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-967fac\"/\"default-dockercfg-crb6m\"" Oct 30 16:35:01.097234 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.097205 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j"] Oct 30 16:35:01.144106 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.144076 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-967fac\"/\"kube-root-ca.crt\"" Oct 30 16:35:01.144285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.144122 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-967fac\"/\"openshift-service-ca.crt\"" Oct 30 16:35:01.184945 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.184908 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-home\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.185148 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.184974 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-dshm\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.185148 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.185104 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24x4q\" (UniqueName: \"kubernetes.io/projected/3db653e1-2db6-485f-a401-df168bd9c785-kube-api-access-24x4q\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.185273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.185180 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-tmp-dir\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.185273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.185238 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3db653e1-2db6-485f-a401-df168bd9c785-tls-certs\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.185368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.185348 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-model-cache\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.286828 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.286793 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-tmp-dir\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.287031 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.286839 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3db653e1-2db6-485f-a401-df168bd9c785-tls-certs\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.287031 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.286881 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-model-cache\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.287031 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.286932 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-home\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.287231 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.286970 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-dshm\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.287231 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.287128 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24x4q\" (UniqueName: \"kubernetes.io/projected/3db653e1-2db6-485f-a401-df168bd9c785-kube-api-access-24x4q\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.287339 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.287273 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-tmp-dir\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.287396 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.287359 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-model-cache\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.287589 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.287568 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-home\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.290572 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.290524 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-dshm\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.290724 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.290694 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3db653e1-2db6-485f-a401-df168bd9c785-tls-certs\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.313027 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.312939 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24x4q\" (UniqueName: \"kubernetes.io/projected/3db653e1-2db6-485f-a401-df168bd9c785-kube-api-access-24x4q\") pod \"simulator-w5-967fac-kserve-6ccc68c76b-wqg5j\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.384257 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.384225 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:01.682472 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:01.682439 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j"] Oct 30 16:35:01.684096 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:35:01.684063 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3db653e1_2db6_485f_a401_df168bd9c785.slice/crio-4a79ce0c13d4cf13736ef2c0dc2d09d985cbae3ff0f66c26576c51a9cb6dcab9 WatchSource:0}: Error finding container 4a79ce0c13d4cf13736ef2c0dc2d09d985cbae3ff0f66c26576c51a9cb6dcab9: Status 404 returned error can't find the container with id 4a79ce0c13d4cf13736ef2c0dc2d09d985cbae3ff0f66c26576c51a9cb6dcab9 Oct 30 16:35:02.111235 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:02.111186 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" event={"ID":"3db653e1-2db6-485f-a401-df168bd9c785","Type":"ContainerStarted","Data":"584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250"} Oct 30 16:35:02.111235 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:02.111230 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" event={"ID":"3db653e1-2db6-485f-a401-df168bd9c785","Type":"ContainerStarted","Data":"4a79ce0c13d4cf13736ef2c0dc2d09d985cbae3ff0f66c26576c51a9cb6dcab9"} Oct 30 16:35:02.150149 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:02.150092 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podStartSLOduration=1.150079575 podStartE2EDuration="1.150079575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:02.149043564 +0000 UTC m=+811.299324556" watchObservedRunningTime="2026-10-30 16:35:02.150079575 +0000 UTC m=+811.300360563" Oct 30 16:35:02.384566 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:02.384490 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:03.159482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:03.159451 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:35:03.172012 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:03.171987 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:35:04.133494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:04.133461 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:35:07.490798 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.490769 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf"] Oct 30 16:35:07.494716 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.494687 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.519664 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.519637 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a8c22d\"/\"default-dockercfg-cmqzx\"" Oct 30 16:35:07.519664 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.519637 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a8c22d\"/\"simulator-w1-a8c22d-kserve-self-signed-certs\"" Oct 30 16:35:07.519873 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.519672 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a8c22d\"/\"kube-root-ca.crt\"" Oct 30 16:35:07.519873 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.519769 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-a8c22d\"/\"openshift-service-ca.crt\"" Oct 30 16:35:07.522781 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.522759 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf"] Oct 30 16:35:07.653273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.653230 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-home\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.653458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.653280 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzv42\" (UniqueName: \"kubernetes.io/projected/0c87785a-a2ea-4813-afa9-000a42e7de59-kube-api-access-kzv42\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.653458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.653381 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-model-cache\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.653549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.653523 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c87785a-a2ea-4813-afa9-000a42e7de59-tls-certs\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.653635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.653613 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-dshm\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.653694 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.653650 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-tmp-dir\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.754756 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.754633 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-home\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.754953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.754753 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzv42\" (UniqueName: \"kubernetes.io/projected/0c87785a-a2ea-4813-afa9-000a42e7de59-kube-api-access-kzv42\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.754953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.754800 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-model-cache\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.754953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.754913 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c87785a-a2ea-4813-afa9-000a42e7de59-tls-certs\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.755119 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.755001 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-dshm\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.755119 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.755043 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-tmp-dir\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.755397 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.755364 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-model-cache\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.755495 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.755466 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-tmp-dir\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.755700 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.755672 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-home\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.758113 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.758090 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-dshm\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.758275 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.758257 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c87785a-a2ea-4813-afa9-000a42e7de59-tls-certs\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.777747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.777726 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzv42\" (UniqueName: \"kubernetes.io/projected/0c87785a-a2ea-4813-afa9-000a42e7de59-kube-api-access-kzv42\") pod \"simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:07.833025 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:07.832991 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:08.132276 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:08.131434 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf"] Oct 30 16:35:08.134255 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:35:08.134223 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c87785a_a2ea_4813_afa9_000a42e7de59.slice/crio-7ff3226f25be33ecb280102e081811b0f8b3035fe3e6eef0ad8bd69df6e78d34 WatchSource:0}: Error finding container 7ff3226f25be33ecb280102e081811b0f8b3035fe3e6eef0ad8bd69df6e78d34: Status 404 returned error can't find the container with id 7ff3226f25be33ecb280102e081811b0f8b3035fe3e6eef0ad8bd69df6e78d34 Oct 30 16:35:08.148823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:08.148790 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" event={"ID":"0c87785a-a2ea-4813-afa9-000a42e7de59","Type":"ContainerStarted","Data":"7ff3226f25be33ecb280102e081811b0f8b3035fe3e6eef0ad8bd69df6e78d34"} Oct 30 16:35:09.155822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:09.155786 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" event={"ID":"0c87785a-a2ea-4813-afa9-000a42e7de59","Type":"ContainerStarted","Data":"89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61"} Oct 30 16:35:09.193744 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:09.193688 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podStartSLOduration=2.193675164 podStartE2EDuration="2.193675164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:09.193266428 +0000 UTC m=+818.343547416" watchObservedRunningTime="2026-10-30 16:35:09.193675164 +0000 UTC m=+818.343956152" Oct 30 16:35:09.833884 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:09.833848 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:11.384507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:11.384421 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:11.396889 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:11.396863 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:12.186030 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:12.186005 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:35:17.833500 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:17.833461 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:17.847029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:17.847005 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:18.220469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:18.220389 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:35:24.599143 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:35:24.599101 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:35:25.846530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:25.846498 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z"] Oct 30 16:35:25.850174 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:25.850155 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:25.873154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:25.873124 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-750dcb\"/\"premium-w2-750dcb-kserve-self-signed-certs\"" Oct 30 16:35:25.876768 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:25.876741 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z"] Oct 30 16:35:25.946264 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:25.946230 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-model-cache\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:25.946474 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:25.946272 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfn6t\" (UniqueName: \"kubernetes.io/projected/628efd77-1a83-427b-996a-9495a5a74ed3-kube-api-access-kfn6t\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:25.946474 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:25.946431 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-dshm\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:25.946617 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:25.946478 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-tmp-dir\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:25.946617 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:25.946568 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-home\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:25.946617 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:25.946593 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628efd77-1a83-427b-996a-9495a5a74ed3-tls-certs\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.047328 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.047295 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-model-cache\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.047328 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.047334 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfn6t\" (UniqueName: \"kubernetes.io/projected/628efd77-1a83-427b-996a-9495a5a74ed3-kube-api-access-kfn6t\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.047593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.047377 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-dshm\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.047593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.047402 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-tmp-dir\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.047593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.047464 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-home\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.047593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.047493 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628efd77-1a83-427b-996a-9495a5a74ed3-tls-certs\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.047834 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.047771 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-model-cache\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.047889 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.047856 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-tmp-dir\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.048012 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.047992 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-home\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.049837 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.049811 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-dshm\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.050192 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.050174 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628efd77-1a83-427b-996a-9495a5a74ed3-tls-certs\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.070155 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.070129 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfn6t\" (UniqueName: \"kubernetes.io/projected/628efd77-1a83-427b-996a-9495a5a74ed3-kube-api-access-kfn6t\") pod \"premium-w2-750dcb-kserve-667d65c7b7-zd98z\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.166133 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.166044 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:26.473426 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:26.473377 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z"] Oct 30 16:35:26.476752 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:35:26.476719 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod628efd77_1a83_427b_996a_9495a5a74ed3.slice/crio-0fede5c281b0f1f2b96bb282d158d506ff2499768d297fc43e8af01731b4815c WatchSource:0}: Error finding container 0fede5c281b0f1f2b96bb282d158d506ff2499768d297fc43e8af01731b4815c: Status 404 returned error can't find the container with id 0fede5c281b0f1f2b96bb282d158d506ff2499768d297fc43e8af01731b4815c Oct 30 16:35:27.267426 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:27.267384 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" event={"ID":"628efd77-1a83-427b-996a-9495a5a74ed3","Type":"ContainerStarted","Data":"586bf37b7422b3b334c2a4b5c3d2a6374a84404dbd3532256f6bc4345a6c0a73"} Oct 30 16:35:27.267426 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:27.267434 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" event={"ID":"628efd77-1a83-427b-996a-9495a5a74ed3","Type":"ContainerStarted","Data":"0fede5c281b0f1f2b96bb282d158d506ff2499768d297fc43e8af01731b4815c"} Oct 30 16:35:27.307066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:27.307014 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podStartSLOduration=2.3069964929999998 podStartE2EDuration="2.306996493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:27.304538862 +0000 UTC m=+836.454819850" watchObservedRunningTime="2026-10-30 16:35:27.306996493 +0000 UTC m=+836.457277481" Oct 30 16:35:28.166642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:28.166597 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:36.166756 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:36.166723 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:36.180217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:36.180192 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:36.332251 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:36.332223 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:35:43.638562 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.638529 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x"] Oct 30 16:35:43.643535 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.643513 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.670512 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.670472 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-967fac\"/\"premium-w5-967fac-kserve-self-signed-certs\"" Oct 30 16:35:43.672935 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.672910 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x"] Oct 30 16:35:43.733766 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.733733 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-dshm\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.733766 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.733772 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsthq\" (UniqueName: \"kubernetes.io/projected/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-kube-api-access-hsthq\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.734042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.733886 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-model-cache\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.734042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.733951 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-home\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.734152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.734050 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tls-certs\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.734152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.734095 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tmp-dir\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.835334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.835295 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-home\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.835334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.835338 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tls-certs\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.835594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.835365 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tmp-dir\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.835594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.835426 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-dshm\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.835594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.835442 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsthq\" (UniqueName: \"kubernetes.io/projected/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-kube-api-access-hsthq\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.835811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.835784 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-home\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.835992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.835817 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tmp-dir\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.836097 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.835539 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-model-cache\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.836373 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.836135 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-model-cache\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.838376 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.838350 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tls-certs\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.838815 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.838793 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-dshm\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.861404 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.861342 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsthq\" (UniqueName: \"kubernetes.io/projected/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-kube-api-access-hsthq\") pod \"premium-w5-967fac-kserve-5d649b7b9f-xtm4x\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:43.956048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:43.955966 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:44.260762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:44.260735 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x"] Oct 30 16:35:44.262288 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:35:44.262261 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93bb0a4a_4dcb_49d6_98a5_eb537b02bc03.slice/crio-4757cf9f2a5349d2833842bc575b4e91651e76cdf333560d2dda42c01f2fec0f WatchSource:0}: Error finding container 4757cf9f2a5349d2833842bc575b4e91651e76cdf333560d2dda42c01f2fec0f: Status 404 returned error can't find the container with id 4757cf9f2a5349d2833842bc575b4e91651e76cdf333560d2dda42c01f2fec0f Oct 30 16:35:44.367333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:44.367301 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" event={"ID":"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03","Type":"ContainerStarted","Data":"d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c"} Oct 30 16:35:44.367333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:44.367338 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" event={"ID":"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03","Type":"ContainerStarted","Data":"4757cf9f2a5349d2833842bc575b4e91651e76cdf333560d2dda42c01f2fec0f"} Oct 30 16:35:44.956106 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:44.956069 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:48.603626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.603545 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podStartSLOduration=5.603524525 podStartE2EDuration="5.603524525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:44.414164951 +0000 UTC m=+853.564445939" watchObservedRunningTime="2026-10-30 16:35:48.603524525 +0000 UTC m=+857.753805514" Oct 30 16:35:48.606379 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.606356 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8"] Oct 30 16:35:48.616019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.615994 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.638268 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.638242 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8"] Oct 30 16:35:48.657935 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.657904 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-a8c22d\"/\"premium-w1-a8c22d-kserve-self-signed-certs\"" Oct 30 16:35:48.793495 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.793460 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6209d2-228d-4df9-a185-59381ea836a8-tls-certs\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.793680 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.793526 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-home\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.793726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.793688 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-tmp-dir\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.793762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.793734 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-dshm\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.793793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.793760 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-model-cache\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.793924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.793898 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bc8b\" (UniqueName: \"kubernetes.io/projected/7f6209d2-228d-4df9-a185-59381ea836a8-kube-api-access-6bc8b\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.895041 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.894963 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bc8b\" (UniqueName: \"kubernetes.io/projected/7f6209d2-228d-4df9-a185-59381ea836a8-kube-api-access-6bc8b\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.895201 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.895073 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6209d2-228d-4df9-a185-59381ea836a8-tls-certs\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.895201 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.895126 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-home\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.895201 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.895172 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-tmp-dir\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.895201 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.895197 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-dshm\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.895407 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.895219 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-model-cache\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.896482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.895668 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-tmp-dir\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.896482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.895700 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-model-cache\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.896482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.895721 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-home\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.897707 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.897684 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-dshm\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.897801 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.897770 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6209d2-228d-4df9-a185-59381ea836a8-tls-certs\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.920017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.919986 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bc8b\" (UniqueName: \"kubernetes.io/projected/7f6209d2-228d-4df9-a185-59381ea836a8-kube-api-access-6bc8b\") pod \"premium-w1-a8c22d-kserve-865c4784f5-tsgv8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:48.932993 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:48.932963 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:49.235214 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:49.235193 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8"] Oct 30 16:35:49.236989 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:35:49.236963 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f6209d2_228d_4df9_a185_59381ea836a8.slice/crio-aa8bc1f41dfa9ec7f058ce7e11610b9d43fce9fc469ecda77ae52c9aa08d29ce WatchSource:0}: Error finding container aa8bc1f41dfa9ec7f058ce7e11610b9d43fce9fc469ecda77ae52c9aa08d29ce: Status 404 returned error can't find the container with id aa8bc1f41dfa9ec7f058ce7e11610b9d43fce9fc469ecda77ae52c9aa08d29ce Oct 30 16:35:49.399063 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:49.399027 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" event={"ID":"7f6209d2-228d-4df9-a185-59381ea836a8","Type":"ContainerStarted","Data":"20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f"} Oct 30 16:35:49.399239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:49.399079 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" event={"ID":"7f6209d2-228d-4df9-a185-59381ea836a8","Type":"ContainerStarted","Data":"aa8bc1f41dfa9ec7f058ce7e11610b9d43fce9fc469ecda77ae52c9aa08d29ce"} Oct 30 16:35:49.439494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:49.439433 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podStartSLOduration=1.439414293 podStartE2EDuration="1.439414293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:35:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:35:49.436389553 +0000 UTC m=+858.586670542" watchObservedRunningTime="2026-10-30 16:35:49.439414293 +0000 UTC m=+858.589695284" Oct 30 16:35:49.933736 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:49.933693 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:53.957138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:53.957100 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:53.970001 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:53.969976 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:54.441306 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:54.441276 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:35:54.598712 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:35:54.598677 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:35:58.933236 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:58.933195 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:58.946226 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:58.946201 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:35:59.473822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:35:59.473793 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:36:03.899535 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.899500 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj"] Oct 30 16:36:03.902578 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.902553 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:03.916214 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.916193 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-263f00-openshift-default-dockercfg-nr2tc\"" Oct 30 16:36:03.928323 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.928293 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj"] Oct 30 16:36:03.956041 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.956003 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:03.956217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.956050 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:03.956217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.956070 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bt68\" (UniqueName: \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-kube-api-access-5bt68\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:03.956299 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.956264 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:03.956333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.956308 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:03.956401 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.956385 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:03.956515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.956494 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:03.956577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.956528 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:03.956577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:03.956557 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058026 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.057992 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058205 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058036 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058205 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058068 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058205 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058153 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058205 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058190 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058447 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058231 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058447 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058311 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058447 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058361 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058447 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058393 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bt68\" (UniqueName: \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-kube-api-access-5bt68\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058546 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058813 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.058917 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.058882 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.059046 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.059031 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.059114 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.059090 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.060591 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.060570 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.061184 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.061164 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.080567 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.080540 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.080782 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.080759 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bt68\" (UniqueName: \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-kube-api-access-5bt68\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.215655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.215560 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:04.530951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.530913 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj"] Oct 30 16:36:04.532461 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:36:04.532428 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4350912f_cfe4_4ca9_8e9e_43dc41b25cf4.slice/crio-0c69b5261872597e09ba42b298817daa6d289f44c9e891067dcde2635e21c1b4 WatchSource:0}: Error finding container 0c69b5261872597e09ba42b298817daa6d289f44c9e891067dcde2635e21c1b4: Status 404 returned error can't find the container with id 0c69b5261872597e09ba42b298817daa6d289f44c9e891067dcde2635e21c1b4 Oct 30 16:36:04.534930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.534891 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:04.534995 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.534957 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:04.534995 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:04.534985 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:05.509212 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:05.509170 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" event={"ID":"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4","Type":"ContainerStarted","Data":"646e311d360a39208f35e7bb38354e7c68b7bc201a96a0a7466be816d50ea8b3"} Oct 30 16:36:05.509675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:05.509244 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" event={"ID":"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4","Type":"ContainerStarted","Data":"0c69b5261872597e09ba42b298817daa6d289f44c9e891067dcde2635e21c1b4"} Oct 30 16:36:05.579166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:05.579105 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" podStartSLOduration=2.579086371 podStartE2EDuration="2.579086371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:05.578117144 +0000 UTC m=+874.728398134" watchObservedRunningTime="2026-10-30 16:36:05.579086371 +0000 UTC m=+874.729367361" Oct 30 16:36:06.226629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:06.224507 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:06.235768 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:06.235731 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:36:06.237060 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:06.237025 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 16:36:07.234749 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:07.234713 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:36:07.235266 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:07.234780 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 16:36:08.216758 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:08.216724 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:36:08.217050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:08.217022 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 16:36:09.222066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:09.222024 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:36:09.222709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:09.222674 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 16:36:10.219791 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:10.219753 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:36:10.220081 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:10.220051 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 16:36:12.061547 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:12.061513 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:12.062031 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:12.061826 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:12.062923 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:12.062904 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:36:18.533146 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.533110 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4"] Oct 30 16:36:18.536444 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.536421 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.549252 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.549228 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-0b2348-openshift-default-dockercfg-86t2v\"" Oct 30 16:36:18.563496 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.563466 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4"] Oct 30 16:36:18.636433 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.636403 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.636661 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.636471 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.636661 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.636597 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.636772 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.636730 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.636772 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.636761 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.636886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.636779 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lpcb\" (UniqueName: \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-kube-api-access-6lpcb\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.636886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.636853 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.636965 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.636924 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.637006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.636980 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.738498 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.738446 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.738498 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.738507 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.738802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.738534 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lpcb\" (UniqueName: \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-kube-api-access-6lpcb\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.738802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.738567 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.738802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.738652 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.738802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.738689 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.738802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.738718 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.739058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.738817 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.739058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.738855 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.739058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.738954 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.739226 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.739067 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.739226 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.739095 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.739226 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.739176 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.739555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.739534 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.740986 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.740964 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.741354 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.741336 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.761029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.761002 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.761319 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.761304 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lpcb\" (UniqueName: \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-kube-api-access-6lpcb\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:18.850586 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:18.850467 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:19.165838 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:19.165808 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4"] Oct 30 16:36:19.167009 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:36:19.166979 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae9ef48_3347_49a8_b8c9_8a8634d083c1.slice/crio-0e54e019b77cd7f905e85528d964dfe1b29bf09698382552f9ff9b78e43b67a4 WatchSource:0}: Error finding container 0e54e019b77cd7f905e85528d964dfe1b29bf09698382552f9ff9b78e43b67a4: Status 404 returned error can't find the container with id 0e54e019b77cd7f905e85528d964dfe1b29bf09698382552f9ff9b78e43b67a4 Oct 30 16:36:19.169284 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:19.169254 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:19.169364 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:19.169314 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:19.169364 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:19.169343 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:19.637058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:19.636941 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" event={"ID":"0ae9ef48-3347-49a8-b8c9-8a8634d083c1","Type":"ContainerStarted","Data":"8a8f273f36689061ad36b51c4b8fad050c198f1dce2d3b539897442af74236a4"} Oct 30 16:36:19.637485 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:19.637093 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" event={"ID":"0ae9ef48-3347-49a8-b8c9-8a8634d083c1","Type":"ContainerStarted","Data":"0e54e019b77cd7f905e85528d964dfe1b29bf09698382552f9ff9b78e43b67a4"} Oct 30 16:36:19.679153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:19.679091 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" podStartSLOduration=1.679071671 podStartE2EDuration="1.679071671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:19.677961424 +0000 UTC m=+888.828242412" watchObservedRunningTime="2026-10-30 16:36:19.679071671 +0000 UTC m=+888.829352672" Oct 30 16:36:19.850891 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:19.850856 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:20.806815 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:20.806780 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r"] Oct 30 16:36:20.811634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:20.809659 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:20.839424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:20.836895 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r"] Oct 30 16:36:20.843042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:20.843021 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:36:20.843258 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:20.843242 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:36:20.852258 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:20.852233 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 16:36:20.852391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:20.852285 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 16:36:20.968409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:20.968369 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45e6fe7-dxm7r\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:20.968624 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:20.968445 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnfmb\" (UniqueName: \"kubernetes.io/projected/9da0e5b6-71e3-47cc-adee-e98bd2013137-kube-api-access-pnfmb\") pod \"maas-api-revoke-sub-a45e6fe7-dxm7r\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:20.968624 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:20.968491 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45e6fe7-dxm7r\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:21.069883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.069793 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45e6fe7-dxm7r\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:21.069883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.069872 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnfmb\" (UniqueName: \"kubernetes.io/projected/9da0e5b6-71e3-47cc-adee-e98bd2013137-kube-api-access-pnfmb\") pod \"maas-api-revoke-sub-a45e6fe7-dxm7r\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:21.070075 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.069919 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45e6fe7-dxm7r\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:21.070629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.070576 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a45e6fe7-dxm7r\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:21.072409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.072391 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a45e6fe7-dxm7r\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:21.091617 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.091570 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnfmb\" (UniqueName: \"kubernetes.io/projected/9da0e5b6-71e3-47cc-adee-e98bd2013137-kube-api-access-pnfmb\") pod \"maas-api-revoke-sub-a45e6fe7-dxm7r\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:21.144233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.144200 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:21.444889 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.444854 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r"] Oct 30 16:36:21.446088 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:36:21.446058 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9da0e5b6_71e3_47cc_adee_e98bd2013137.slice/crio-30fb54436ca5b4c887686368bab44bb8ae93252dcd60b6dcda89c82d89126630 WatchSource:0}: Error finding container 30fb54436ca5b4c887686368bab44bb8ae93252dcd60b6dcda89c82d89126630: Status 404 returned error can't find the container with id 30fb54436ca5b4c887686368bab44bb8ae93252dcd60b6dcda89c82d89126630 Oct 30 16:36:21.448107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.448088 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:36:21.650104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.650061 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" event={"ID":"9da0e5b6-71e3-47cc-adee-e98bd2013137","Type":"ContainerStarted","Data":"30fb54436ca5b4c887686368bab44bb8ae93252dcd60b6dcda89c82d89126630"} Oct 30 16:36:21.851852 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.851817 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 16:36:21.852349 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:21.851877 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 16:36:22.851224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:22.851184 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 16:36:22.851414 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:22.851244 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 16:36:23.851432 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:23.851391 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 16:36:23.851946 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:23.851478 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 16:36:24.597830 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:36:24.597795 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:36:24.671959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:24.671920 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" event={"ID":"9da0e5b6-71e3-47cc-adee-e98bd2013137","Type":"ContainerStarted","Data":"d1992c4d3855c091b708330a9505c50fe9fa96f0711bf8a1e9630cbbac6bfbfc"} Oct 30 16:36:24.713701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:24.709703 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" podStartSLOduration=2.089981375 podStartE2EDuration="4.709685753s" totalImagesPullingTime="2.619704378s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:24.709086345 +0000 UTC m=+893.859367334" watchObservedRunningTime="2026-10-30 16:36:24.709685753 +0000 UTC m=+893.859966742" Oct 30 16:36:24.851050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:24.851016 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 30 16:36:24.851208 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:24.851067 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 30 16:36:25.680969 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:25.680914 6400 generic.go:356] "Generic (PLEG): container finished" podID="9da0e5b6-71e3-47cc-adee-e98bd2013137" containerID="d1992c4d3855c091b708330a9505c50fe9fa96f0711bf8a1e9630cbbac6bfbfc" exitCode=0 Oct 30 16:36:25.681423 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:25.681000 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" event={"ID":"9da0e5b6-71e3-47cc-adee-e98bd2013137","Type":"ContainerDied","Data":"d1992c4d3855c091b708330a9505c50fe9fa96f0711bf8a1e9630cbbac6bfbfc"} Oct 30 16:36:25.854836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:25.854806 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:25.855144 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:25.855106 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:25.855788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:25.855768 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:36:26.998336 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:26.998305 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:27.095515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.095483 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-xks-ca\") pod \"9da0e5b6-71e3-47cc-adee-e98bd2013137\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " Oct 30 16:36:27.095819 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.095526 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9da0e5b6-71e3-47cc-adee-e98bd2013137-kube-api-access-pnfmb\" (UniqueName: \"kubernetes.io/projected/9da0e5b6-71e3-47cc-adee-e98bd2013137-kube-api-access-pnfmb\") pod \"9da0e5b6-71e3-47cc-adee-e98bd2013137\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " Oct 30 16:36:27.095819 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.095727 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-openshift-ca\") pod \"9da0e5b6-71e3-47cc-adee-e98bd2013137\" (UID: \"9da0e5b6-71e3-47cc-adee-e98bd2013137\") " Oct 30 16:36:27.096238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.096200 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-openshift-ca" pod "9da0e5b6-71e3-47cc-adee-e98bd2013137" (UID: "9da0e5b6-71e3-47cc-adee-e98bd2013137"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:36:27.098696 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.098658 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9da0e5b6-71e3-47cc-adee-e98bd2013137-kube-api-access-pnfmb" pod "9da0e5b6-71e3-47cc-adee-e98bd2013137" (UID: "9da0e5b6-71e3-47cc-adee-e98bd2013137"). InnerVolumeSpecName "kube-api-access-pnfmb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:36:27.098941 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.098919 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-xks-ca" pod "9da0e5b6-71e3-47cc-adee-e98bd2013137" (UID: "9da0e5b6-71e3-47cc-adee-e98bd2013137"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:36:27.197095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.197058 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:36:27.197095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.197089 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9da0e5b6-71e3-47cc-adee-e98bd2013137-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:36:27.197095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.197098 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnfmb\" (UniqueName: \"kubernetes.io/projected/9da0e5b6-71e3-47cc-adee-e98bd2013137-kube-api-access-pnfmb\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:36:27.694197 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.694170 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" Oct 30 16:36:27.694372 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.694207 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r" event={"ID":"9da0e5b6-71e3-47cc-adee-e98bd2013137","Type":"ContainerDied","Data":"30fb54436ca5b4c887686368bab44bb8ae93252dcd60b6dcda89c82d89126630"} Oct 30 16:36:27.694372 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:27.694249 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="30fb54436ca5b4c887686368bab44bb8ae93252dcd60b6dcda89c82d89126630" Oct 30 16:36:33.594934 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.594900 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p"] Oct 30 16:36:33.596402 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.596363 6400 cpu_manager.go:414] "containerMap: removing container" podUID="9da0e5b6-71e3-47cc-adee-e98bd2013137" containerName="revoke-keys" Oct 30 16:36:33.596568 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.596553 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9da0e5b6-71e3-47cc-adee-e98bd2013137" containerName="revoke-keys" Oct 30 16:36:33.596830 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.596806 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9da0e5b6-71e3-47cc-adee-e98bd2013137" containerName="revoke-keys" Oct 30 16:36:33.646458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.646423 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p"] Oct 30 16:36:33.646739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.646524 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.767654 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.767611 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.767868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.767736 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.767868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.767790 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.767868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.767818 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psmnw\" (UniqueName: \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-kube-api-access-psmnw\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.768054 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.767942 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.768054 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.768029 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.768151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.768089 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.768151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.768127 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.768249 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.768161 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869221 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869135 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869221 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869189 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869221 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869209 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869226 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psmnw\" (UniqueName: \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-kube-api-access-psmnw\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869258 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869439 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869515 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869560 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869653 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869661 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869670 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.869881 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869723 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.870029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.869998 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.870139 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.870097 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.872040 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.872014 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.872152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.872137 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.890793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.890760 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.890943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.890795 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psmnw\" (UniqueName: \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-kube-api-access-psmnw\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:33.963939 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:33.963910 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:34.281842 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:34.281786 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p"] Oct 30 16:36:34.282372 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:36:34.282344 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd90b88ae_d012_4128_9479_ed9f5e8bbff2.slice/crio-a89a680b355073a95ebc05ab74c17ed281f3df8a050d2a6d610ddf085d24503e WatchSource:0}: Error finding container a89a680b355073a95ebc05ab74c17ed281f3df8a050d2a6d610ddf085d24503e: Status 404 returned error can't find the container with id a89a680b355073a95ebc05ab74c17ed281f3df8a050d2a6d610ddf085d24503e Oct 30 16:36:34.284991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:34.284951 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:34.285073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:34.285023 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:34.285073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:34.285052 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:34.739173 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:34.739136 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" event={"ID":"d90b88ae-d012-4128-9479-ed9f5e8bbff2","Type":"ContainerStarted","Data":"cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2"} Oct 30 16:36:34.739577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:34.739203 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" event={"ID":"d90b88ae-d012-4128-9479-ed9f5e8bbff2","Type":"ContainerStarted","Data":"a89a680b355073a95ebc05ab74c17ed281f3df8a050d2a6d610ddf085d24503e"} Oct 30 16:36:34.799520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:34.799465 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" podStartSLOduration=1.7994517669999999 podStartE2EDuration="1.799451767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:34.789028131 +0000 UTC m=+903.939309123" watchObservedRunningTime="2026-10-30 16:36:34.799451767 +0000 UTC m=+903.949732820" Oct 30 16:36:34.966816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:34.966784 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:35.966408 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:35.966372 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:36:35.966829 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:35.966428 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 16:36:36.964490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:36.964458 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:36:36.964689 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:36.964524 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 16:36:37.964890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:37.964852 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:36:37.965377 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:37.964911 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 16:36:38.975894 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:38.975862 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:36:38.976412 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:38.975954 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 16:36:39.967629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:39.966989 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:36:39.967629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:39.967052 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 16:36:40.965009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:40.964923 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" start-of-body= Oct 30 16:36:40.965717 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:40.965683 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": dial tcp 10.134.0.63:15021: connect: connection refused" Oct 30 16:36:42.436053 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:42.436018 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:42.436539 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:42.436455 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:42.437116 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:42.437095 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 16:36:47.075801 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.075769 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp"] Oct 30 16:36:47.078536 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.078516 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.093674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.093646 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg"] Oct 30 16:36:47.097380 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.097356 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.131856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.131767 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp"] Oct 30 16:36:47.139104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.139079 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg"] Oct 30 16:36:47.178124 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178093 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.178305 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178130 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.178305 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178154 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.178305 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178174 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208934b7-c63d-4074-a8bc-85d627b145df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.178441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178306 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcff\" (UniqueName: \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-kube-api-access-gmcff\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.178441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178334 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.178441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178381 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.178441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178409 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-klsx9\" (UniqueName: \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-kube-api-access-klsx9\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.178441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178427 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.178626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178455 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.178626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178475 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.178626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178553 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.178626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178591 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763ed816-0c8a-4043-ba58-636b1b9f9570-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.178793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178689 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.178793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178720 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208934b7-c63d-4074-a8bc-85d627b145df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.178793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178739 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.178793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178763 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.178936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.178896 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.280121 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280080 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208934b7-c63d-4074-a8bc-85d627b145df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280121 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280124 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.280383 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280142 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280383 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280180 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.280383 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280239 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.280549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280373 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280411 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280436 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208934b7-c63d-4074-a8bc-85d627b145df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280484 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcff\" (UniqueName: \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-kube-api-access-gmcff\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280534 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280571 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.280860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280589 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.280860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280596 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280665 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-klsx9\" (UniqueName: \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-kube-api-access-klsx9\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.280860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280696 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280701 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280758 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.280860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280791 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.280860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280833 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.281548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280868 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763ed816-0c8a-4043-ba58-636b1b9f9570-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.281548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280939 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.281548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280955 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.281548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.281046 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.281548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.280597 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.281548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.281274 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.281548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.281274 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.281922 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.281834 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763ed816-0c8a-4043-ba58-636b1b9f9570-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.281922 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.281878 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208934b7-c63d-4074-a8bc-85d627b145df-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.283961 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.283904 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.284087 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.284040 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.284087 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.284083 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208934b7-c63d-4074-a8bc-85d627b145df-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.284235 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.284217 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.305042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.305013 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcff\" (UniqueName: \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-kube-api-access-gmcff\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.305213 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.305189 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-klsx9\" (UniqueName: \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-kube-api-access-klsx9\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.305722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.305698 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lzfbg\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.306155 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.306128 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5fbpp\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.392484 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.392390 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:47.411903 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.411776 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:47.716210 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.716133 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp"] Oct 30 16:36:47.729716 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:36:47.729689 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod208934b7_c63d_4074_a8bc_85d627b145df.slice/crio-954902855be85b5ed2266160c1f6c27d49580411634feeb1c9245773573dc091 WatchSource:0}: Error finding container 954902855be85b5ed2266160c1f6c27d49580411634feeb1c9245773573dc091: Status 404 returned error can't find the container with id 954902855be85b5ed2266160c1f6c27d49580411634feeb1c9245773573dc091 Oct 30 16:36:47.732137 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.732105 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:47.732229 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.732179 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:47.732229 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.732210 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:47.755507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.755484 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg"] Oct 30 16:36:47.761861 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:36:47.761830 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod763ed816_0c8a_4043_ba58_636b1b9f9570.slice/crio-4fba2bf7049901caa89253453fe81b58b0af5ea5da3027ecb869d80415a06703 WatchSource:0}: Error finding container 4fba2bf7049901caa89253453fe81b58b0af5ea5da3027ecb869d80415a06703: Status 404 returned error can't find the container with id 4fba2bf7049901caa89253453fe81b58b0af5ea5da3027ecb869d80415a06703 Oct 30 16:36:47.764483 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.764449 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:47.764585 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.764539 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:47.764725 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.764584 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:47.846645 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.839157 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" event={"ID":"208934b7-c63d-4074-a8bc-85d627b145df","Type":"ContainerStarted","Data":"d844ec743c3bfa6d72c0972622f807b8ad8ecdb2ffa6d1f57e6da1c628c6c016"} Oct 30 16:36:47.846645 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.839219 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" event={"ID":"208934b7-c63d-4074-a8bc-85d627b145df","Type":"ContainerStarted","Data":"954902855be85b5ed2266160c1f6c27d49580411634feeb1c9245773573dc091"} Oct 30 16:36:47.846645 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.841543 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" event={"ID":"763ed816-0c8a-4043-ba58-636b1b9f9570","Type":"ContainerStarted","Data":"d4a479681567857b098df8ac71c8b0d1a3cfb29e0aaf01519184b91a8fa8a8c8"} Oct 30 16:36:47.846645 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.841592 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" event={"ID":"763ed816-0c8a-4043-ba58-636b1b9f9570","Type":"ContainerStarted","Data":"4fba2bf7049901caa89253453fe81b58b0af5ea5da3027ecb869d80415a06703"} Oct 30 16:36:47.884029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.883952 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podStartSLOduration=0.883933436 podStartE2EDuration="883.933436ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:47.881767142 +0000 UTC m=+917.032048132" watchObservedRunningTime="2026-10-30 16:36:47.883933436 +0000 UTC m=+917.034214425" Oct 30 16:36:47.929199 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:47.929130 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podStartSLOduration=0.92910926 podStartE2EDuration="929.10926ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:47.92569617 +0000 UTC m=+917.075977158" watchObservedRunningTime="2026-10-30 16:36:47.92910926 +0000 UTC m=+917.079390328" Oct 30 16:36:48.307506 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.307471 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j"] Oct 30 16:36:48.314143 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.314111 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.335073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.335040 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-750dcb-openshift-default-dockercfg-fbrrg\"" Oct 30 16:36:48.360142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.359988 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j"] Oct 30 16:36:48.393176 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.393149 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:36:48.394519 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.394491 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:48.394654 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.394553 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:48.400642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.400594 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.400760 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.400715 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.400828 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.400765 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35adff5a-315a-4e33-922c-ec81f3091a8f-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.400828 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.400786 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64rtl\" (UniqueName: \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-kube-api-access-64rtl\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.400828 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.400801 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.400959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.400945 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.401018 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.401007 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.401060 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.401032 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.401153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.401136 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.433627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.426971 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:36:48.455698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.446204 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:48.455698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.446280 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:48.502416 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502378 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502416 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502418 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502442 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502478 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502516 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502557 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35adff5a-315a-4e33-922c-ec81f3091a8f-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502575 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64rtl\" (UniqueName: \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-kube-api-access-64rtl\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502591 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502669 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502922 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502792 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502955 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502925 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.502995 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.502961 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.503134 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.503118 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.503384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.503357 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35adff5a-315a-4e33-922c-ec81f3091a8f-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.505514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.505494 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.505681 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.505662 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.562691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.558776 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64rtl\" (UniqueName: \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-kube-api-access-64rtl\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.569627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.569011 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.674304 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.674274 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:48.766431 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.766394 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x"] Oct 30 16:36:48.770975 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.770454 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.790452 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.790419 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x"] Oct 30 16:36:48.793106 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.792919 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-d693d0-openshift-default-dockercfg-qggxm\"" Oct 30 16:36:48.805175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.805143 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.805297 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.805191 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.805354 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.805309 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.805354 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.805344 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.805432 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.805379 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.805481 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.805467 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.805535 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.805500 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.805576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.805553 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.805633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.805581 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7q76\" (UniqueName: \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-kube-api-access-n7q76\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.907325 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.907239 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.907325 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.907277 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.907504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.907336 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.907504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.907369 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.907504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.907413 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.907504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.907433 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.907504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.907452 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.907504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.907489 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.907728 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.907508 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7q76\" (UniqueName: \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-kube-api-access-n7q76\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.909291 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.909268 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.957631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.957539 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.958716 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.958693 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.972331 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.959048 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.972592 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.968275 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.972748 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.969648 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj"] Oct 30 16:36:48.973404 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.973360 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7q76\" (UniqueName: \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-kube-api-access-n7q76\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.974033 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.973984 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.974544 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.974519 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.976420 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.976379 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:48.978840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.978822 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj"] Oct 30 16:36:48.980858 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.980787 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:48.994551 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:48.994530 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-967fac-openshift-default-dockercfg-lfjgc\"" Oct 30 16:36:49.008469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.008439 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.008671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.008475 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.008671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.008502 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.008671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.008618 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.008863 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.008681 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2xwv\" (UniqueName: \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-kube-api-access-v2xwv\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.008863 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.008743 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.008863 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.008836 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.009013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.008898 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bb001a-9638-4cd5-a836-fb2e2cfad946-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.009013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.008933 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.078239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.078141 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.078349 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.078237 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.078349 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.078282 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.079222 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.079199 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j"] Oct 30 16:36:49.089434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.083975 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2"] Oct 30 16:36:49.091543 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.091508 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2"] Oct 30 16:36:49.091678 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.091667 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.091773 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.091753 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:49.134650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.109975 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.140853 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.140824 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.145760 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.145722 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bb001a-9638-4cd5-a836-fb2e2cfad946-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.149726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.149693 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.153703 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.153667 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.162685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.153925 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.162972 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.162942 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99286a4-40e3-405c-9e7c-4ca6851357da-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.163159 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.163141 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.163289 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.163275 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.163394 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.163381 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.163511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.163498 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.163655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.163639 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.163818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.163797 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.163921 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.163907 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.166650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.164000 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.166650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.164025 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.166650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.153102 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.166650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.153779 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bb001a-9638-4cd5-a836-fb2e2cfad946-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.166650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.164311 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.166650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.164390 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.166650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.164531 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2xwv\" (UniqueName: \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-kube-api-access-v2xwv\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.166650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.164699 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.166650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.164776 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lb6r\" (UniqueName: \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-kube-api-access-9lb6r\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.169298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.169273 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.173378 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.173313 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.188592 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.188559 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.189219 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.189194 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2xwv\" (UniqueName: \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-kube-api-access-v2xwv\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.266433 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.266384 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99286a4-40e3-405c-9e7c-4ca6851357da-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.266670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.266486 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.266670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.266531 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.266670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.266585 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.266670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.266630 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.266936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.266718 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lb6r\" (UniqueName: \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-kube-api-access-9lb6r\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.266936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.266779 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.266936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.266873 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.267094 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.266935 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.268489 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.268427 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.268756 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.268645 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.268890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.268756 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.268890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.268773 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99286a4-40e3-405c-9e7c-4ca6851357da-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.269005 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.268885 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.272505 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.272463 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.272769 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.272669 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.291041 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.290951 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.291789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.291762 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lb6r\" (UniqueName: \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-kube-api-access-9lb6r\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.297109 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.297088 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:49.393794 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.393329 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:49.393794 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.393386 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:49.415163 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.414984 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:49.415163 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.415040 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:49.474627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.470859 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:49.572662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.572391 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x"] Oct 30 16:36:49.582900 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:36:49.582059 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca64f363_9b91_46d1_8e52_f35df4dc1a3b.slice/crio-6683d7c3be57ba29c5d79a55bc0c1c2ec0b885db56f8dddb6d3d7e1c735b29f5 WatchSource:0}: Error finding container 6683d7c3be57ba29c5d79a55bc0c1c2ec0b885db56f8dddb6d3d7e1c735b29f5: Status 404 returned error can't find the container with id 6683d7c3be57ba29c5d79a55bc0c1c2ec0b885db56f8dddb6d3d7e1c735b29f5 Oct 30 16:36:49.586953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.585746 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.586953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.585849 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.586953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.585894 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.767490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.767453 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj"] Oct 30 16:36:49.783323 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.783275 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.783466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.783363 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.783466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.783396 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:49.888312 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.888235 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" event={"ID":"07bb001a-9638-4cd5-a836-fb2e2cfad946","Type":"ContainerStarted","Data":"37c841013e9a338f5a25008c6c70996b7f5270b86c7194d975b2c18ad0d087f1"} Oct 30 16:36:49.892352 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.891188 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" event={"ID":"ca64f363-9b91-46d1-8e52-f35df4dc1a3b","Type":"ContainerStarted","Data":"e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d"} Oct 30 16:36:49.892352 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.891246 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" event={"ID":"ca64f363-9b91-46d1-8e52-f35df4dc1a3b","Type":"ContainerStarted","Data":"6683d7c3be57ba29c5d79a55bc0c1c2ec0b885db56f8dddb6d3d7e1c735b29f5"} Oct 30 16:36:49.903656 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.901951 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" event={"ID":"35adff5a-315a-4e33-922c-ec81f3091a8f","Type":"ContainerStarted","Data":"220dff0781bdf872793c1ddfeb32a95e6d0a472c715bea50838860a4b108ffad"} Oct 30 16:36:49.906251 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.906219 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" event={"ID":"35adff5a-315a-4e33-922c-ec81f3091a8f","Type":"ContainerStarted","Data":"b5852fb2abaa346881f7493f14d4655571bd7beeb25f59158921692c5277bfe3"} Oct 30 16:36:49.954629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:49.947805 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podStartSLOduration=1.947784531 podStartE2EDuration="1.947784531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:49.945209352 +0000 UTC m=+919.095490340" watchObservedRunningTime="2026-10-30 16:36:49.947784531 +0000 UTC m=+919.098065519" Oct 30 16:36:50.078216 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.076926 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:50.078216 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.077099 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:50.078216 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.077195 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:36:50.079025 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.078991 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2"] Oct 30 16:36:50.085153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.083632 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podStartSLOduration=2.083581536 podStartE2EDuration="2.083581536s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:49.988661399 +0000 UTC m=+919.138942386" watchObservedRunningTime="2026-10-30 16:36:50.083581536 +0000 UTC m=+919.233862526" Oct 30 16:36:50.094823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.094792 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:36:50.096022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.095995 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.096113 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.096044 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:36:50.395055 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.395018 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.395654 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.395620 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:50.422638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.421817 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.422638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.421873 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:50.679633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.677757 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:36:50.680457 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.679891 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:36:50.680457 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:50.679979 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:36:51.101044 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.100159 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podStartSLOduration=2.100140263 podStartE2EDuration="2.100140263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:51.099364955 +0000 UTC m=+920.249645947" watchObservedRunningTime="2026-10-30 16:36:51.100140263 +0000 UTC m=+920.250421251" Oct 30 16:36:51.124628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.106430 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" event={"ID":"07bb001a-9638-4cd5-a836-fb2e2cfad946","Type":"ContainerStarted","Data":"e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d"} Oct 30 16:36:51.124628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.106495 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" event={"ID":"c99286a4-40e3-405c-9e7c-4ca6851357da","Type":"ContainerStarted","Data":"7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8"} Oct 30 16:36:51.124628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.106523 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" event={"ID":"c99286a4-40e3-405c-9e7c-4ca6851357da","Type":"ContainerStarted","Data":"08e8493eacdfa64a94a802b23fbd723106d9106cee8e7f63fd2df0a755e1d90b"} Oct 30 16:36:51.124628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.110882 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.124628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.110932 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:36:51.124628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.120798 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podStartSLOduration=3.120778495 podStartE2EDuration="3.120778495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:36:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:36:51.05394652 +0000 UTC m=+920.204227512" watchObservedRunningTime="2026-10-30 16:36:51.120778495 +0000 UTC m=+920.271059485" Oct 30 16:36:51.302627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.302225 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:36:51.302627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.302360 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.302627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.302403 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:36:51.407686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.407089 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.407686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.407164 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:51.431626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.425992 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.431626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.426051 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:51.489624 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.488669 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:36:51.489624 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.488791 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.489624 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.488838 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:36:51.506664 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.503121 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:51.708736 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.683562 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:36:51.708991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.708958 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:36:51.910213 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.910172 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:51.997760 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:51.997711 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:52.072748 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.072700 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:52.096446 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.096394 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.097204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.096713 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:36:52.114476 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.114434 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 16:36:52.130983 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.130922 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:36:52.220236 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.220185 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:52.220661 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.220118 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:52.382923 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.382817 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.383193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.383149 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:36:52.398745 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.385680 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 16:36:52.412840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.412774 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:52.468629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.468088 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.468629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.468147 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:52.470048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.469006 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.470048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.469059 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:52.470048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.469142 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:52.483096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.483049 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.483400 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.483348 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:36:52.508877 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.508834 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:52.509526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.509504 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:36:52.678684 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.677999 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:36:52.678684 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.678061 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:36:52.912625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.910716 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:52.912625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:52.911503 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:36:53.036640 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.023863 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 16:36:53.066432 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.066398 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:36:53.080771 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.080727 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 16:36:53.091481 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.081520 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:36:53.220626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.115756 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:53.224627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.220992 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 16:36:53.225188 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.119408 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.228706 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.228674 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:36:53.254629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.229121 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:36:53.254991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.254966 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:36:53.257974 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.243982 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:53.273629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.165817 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:53.275303 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.274902 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:36:53.275540 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.275527 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:36:53.303637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.303530 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.303900 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.303875 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:36:53.415430 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.415386 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:53.423869 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.423029 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:36:53.442978 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.442946 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.476747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.476700 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:53.477006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.443204 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.477166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.477144 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:53.479147 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.479118 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.480269 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.480150 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:36:53.480724 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.480707 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:36:53.582143 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.582041 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:36:53.836689 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.649406 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:36:53.837363 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.723029 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:36:53.837474 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.837412 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:36:53.915758 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:53.915712 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:54.136289 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.101524 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 16:36:54.136743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.136683 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:54.172100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.172046 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:54.201674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.197086 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.213989 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.213692 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:36:54.240472 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.240435 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 16:36:54.294813 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.294762 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 16:36:54.361643 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.357168 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.392976 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.392894 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:36:54.393281 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.378740 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:36:54.399054 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.394729 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.399054 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.394775 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:54.413178 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.413143 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.413472 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.413445 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:54.435865 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.432737 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:54.437326 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.437292 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:54.480632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.480576 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:54.481798 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.481772 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.481980 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.481961 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:36:54.675196 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.675024 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:36:54.675196 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.675084 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:36:54.789062 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:36:54.789012 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:36:54.837772 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.837721 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:36:54.918753 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:54.918691 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 16:36:55.094018 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.093981 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.094276 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.094252 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:36:55.104263 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.104225 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.140616 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.140550 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.176637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.173695 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.245625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.244700 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.299626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.299020 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.299626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.299082 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:36:55.299626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.299503 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.400682 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.400191 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:36:55.405638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.402506 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.405638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.402549 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:55.422695 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.419858 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.422695 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.419938 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:55.447628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.445436 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 16:36:55.448125 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.447893 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.478631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.473947 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.478631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.474021 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:36:55.503629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.482716 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.675690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.675025 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:36:55.675690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.675087 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:36:55.839630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.839154 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.848636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.846526 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.878629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.876024 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:55.920631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:55.920097 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 16:36:56.095623 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.094965 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:56.095623 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.095027 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:36:56.117285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.106702 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:56.149627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.142686 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:56.246630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.245715 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:56.299629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.299051 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:56.299629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.299127 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:36:56.299899 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.299645 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 16:36:56.325628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.325424 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:36:56.396016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.395898 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:56.396352 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.396297 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:56.403792 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.403684 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:56.415120 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.415006 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:56.415120 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.415059 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:56.463669 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.463632 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:56.592667 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.472124 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:56.592667 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.485691 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:56.592667 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.511508 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:36:56.592667 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.567754 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:56.716681 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.716447 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:36:56.725835 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.716534 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:36:56.842094 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.841693 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:36:56.984692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:56.983209 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:57.109627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.107856 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:57.109627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.107920 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:36:57.109627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.108427 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:57.150630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.142963 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:57.259681 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.259280 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 16:36:57.321629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.321028 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:57.321629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.321096 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:36:57.394630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.393622 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:57.394630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.393685 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:57.406431 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.404537 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:36:57.417628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.414542 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:57.417628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.414646 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:57.492097 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.492062 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:57.493065 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.492684 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:36:57.518705 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.515818 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:36:57.560221 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.559693 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:57.574422 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.573685 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:57.661444 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.661416 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:36:57.677290 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.677248 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:36:57.677519 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.677498 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:36:57.783466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.782712 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:57.884558 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.884515 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:57.908504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.908475 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:36:57.990724 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:57.990685 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:58.093807 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.093719 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:58.094043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.094018 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:36:58.110205 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.110156 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 16:36:58.149085 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.149041 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:58.357694 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.332594 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 16:36:58.361673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.358773 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:58.361939 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.361895 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:36:58.410184 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.410136 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:58.448055 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.448021 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:58.451743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.451704 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:58.458507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.456163 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body= Oct 30 16:36:58.458507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.456211 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" Oct 30 16:36:58.480627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.478111 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:58.480627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.478168 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:36:58.556626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.556154 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:36:58.581626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.561337 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:58.589623 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.584745 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:58.629948 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.629862 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:36:58.838629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.837633 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:36:58.839953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.839923 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:36:58.840182 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.840156 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:36:58.935719 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.914486 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:36:58.960789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:58.960751 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": context deadline exceeded" Oct 30 16:36:59.119962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.119927 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:36:59.120268 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.120240 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:36:59.121100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.121069 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:59.604628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.603994 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:59.677005 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.676766 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 30 16:36:59.677005 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.676864 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 30 16:36:59.677427 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.677401 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:36:59.678435 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.678405 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:36:59.678930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.678908 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:36:59.679130 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.679096 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:36:59.720631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.707279 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:36:59.720631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.718289 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:36:59.720631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.718367 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:36:59.724462 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.724416 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:59.962679 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.959536 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:36:59.998001 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:36:59.997715 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:00.040868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.039727 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:37:00.102638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.092537 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:37:00.102919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.102885 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:37:00.125221 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.125182 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 16:37:00.198815 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.198767 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 16:37:00.262751 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.262708 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:00.298059 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.298005 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:37:00.298421 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.298394 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:37:00.317245 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.317202 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:00.472399 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.472363 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:37:00.472698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.472669 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:37:00.500536 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.499758 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:00.606897 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.605694 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:00.680401 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.678998 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:37:00.680401 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.679056 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:37:00.737430 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.725700 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:00.737430 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.726149 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:37:00.767803 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.767761 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:00.767998 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.767827 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:00.971742 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.961854 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:00.971742 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:00.962502 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:37:01.090630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.047562 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:01.164718 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.134043 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:01.221630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.207690 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:01.227686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.222711 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:37:01.227686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.223506 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:37:01.238630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.228106 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 30 16:37:01.238630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.228159 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 30 16:37:01.285628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.276007 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:01.285628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.276262 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:37:01.304765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.304730 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:37:01.305102 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.305040 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:37:01.331656 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.319582 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:01.331656 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.319997 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:37:01.458985 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.458952 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lzfbg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:01.463727 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.463688 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:01.518331 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.518300 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:37:01.518620 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.518573 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:37:01.622673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.616409 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:01.725137 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.725100 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" start-of-body= Oct 30 16:37:01.725408 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.725375 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": dial tcp 10.134.0.66:15021: connect: connection refused" Oct 30 16:37:01.725935 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.725911 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 16:37:01.726595 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.726576 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:37:01.753631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.728160 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:37:01.977518 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.939635 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:01.977898 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:01.977779 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:02.014186 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.004936 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 16:37:02.030184 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.029542 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:37:02.031054 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.031027 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:37:02.038628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.034351 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:02.038628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.034717 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:37:02.111714 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.108961 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:37:02.120569 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.120538 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:37:02.298620 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.298558 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:02.342624 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.308100 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:02.481796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.319819 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:02.481796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.357133 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:37:02.481796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.479654 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:37:02.481796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.479692 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:37:02.481796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.479759 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:37:02.481796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.367669 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body= Oct 30 16:37:02.481796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.480683 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" Oct 30 16:37:02.481796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.416790 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:02.481796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.473252 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body= Oct 30 16:37:02.481796 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.481468 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" Oct 30 16:37:02.612712 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.610944 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:37:02.616830 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.616782 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:02.724273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.724230 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:37:02.730498 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.730467 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:37:02.770687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.760145 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:02.814650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.809726 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:37:02.814650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.809813 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:37:02.814650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.809869 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:37:02.824240 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.824210 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:37:02.826869 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.826840 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:37:02.883210 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.883123 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:37:02.884389 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.884327 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:37:02.888226 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:02.888171 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 16:37:03.035172 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:03.035095 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:37:03.674668 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:03.674631 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:37:03.764857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:03.764792 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:37:03.765626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:03.765455 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:37:03.970284 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:03.970203 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:37:04.049636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:04.049588 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:37:04.222361 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:04.222282 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:37:04.223151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:04.223040 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:37:04.224347 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:04.224303 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:37:04.266574 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:04.266543 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:37:04.267017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:04.266954 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:37:04.267954 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:04.267934 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 16:37:24.596568 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:37:24.596515 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:37:48.817781 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.817750 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h"] Oct 30 16:37:48.821434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.821415 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:48.854971 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.854946 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h"] Oct 30 16:37:48.900284 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.900257 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:48.900430 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.900299 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:48.900430 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.900323 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2nt9\" (UniqueName: \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-kube-api-access-q2nt9\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:48.900525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.900441 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745f06d2-beda-4d0f-8f95-322d0ea28329-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:48.900525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.900481 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:48.900591 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.900543 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:48.900658 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.900622 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:48.900696 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.900672 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:48.900752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:48.900741 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.001688 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.001654 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745f06d2-beda-4d0f-8f95-322d0ea28329-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.001878 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.001712 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.001878 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.001781 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.001878 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.001805 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.001878 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.001836 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.001878 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.001869 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.002160 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.001920 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.002160 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.001983 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.002160 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.002028 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2nt9\" (UniqueName: \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-kube-api-access-q2nt9\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.002448 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.002424 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-certs\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.002448 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.002442 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-data\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.002554 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.002450 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745f06d2-beda-4d0f-8f95-322d0ea28329-istiod-ca-cert\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.002554 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.002476 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-credential-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.002664 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.002649 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-socket\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.004450 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.004423 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-envoy\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.004551 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.004509 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-podinfo\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.024673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.024558 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-token\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.024847 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.024721 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2nt9\" (UniqueName: \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-kube-api-access-q2nt9\") pod \"e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.134658 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.134556 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:49.451751 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.451728 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h"] Oct 30 16:37:49.453076 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:37:49.453049 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod745f06d2_beda_4d0f_8f95_322d0ea28329.slice/crio-9d7ca96d0f6687ded6d76e372e9be44efbcd5bf11653e584803b91916fcab0c3 WatchSource:0}: Error finding container 9d7ca96d0f6687ded6d76e372e9be44efbcd5bf11653e584803b91916fcab0c3: Status 404 returned error can't find the container with id 9d7ca96d0f6687ded6d76e372e9be44efbcd5bf11653e584803b91916fcab0c3 Oct 30 16:37:49.455449 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.455409 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:37:49.455560 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.455492 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:37:49.455560 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:49.455536 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:37:50.382406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:50.382370 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" event={"ID":"745f06d2-beda-4d0f-8f95-322d0ea28329","Type":"ContainerStarted","Data":"6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd"} Oct 30 16:37:50.382918 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:50.382448 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" event={"ID":"745f06d2-beda-4d0f-8f95-322d0ea28329","Type":"ContainerStarted","Data":"9d7ca96d0f6687ded6d76e372e9be44efbcd5bf11653e584803b91916fcab0c3"} Oct 30 16:37:50.447003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:50.446951 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" podStartSLOduration=2.446936593 podStartE2EDuration="2.446936593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:37:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:37:50.436116589 +0000 UTC m=+979.586397578" watchObservedRunningTime="2026-10-30 16:37:50.446936593 +0000 UTC m=+979.597217616" Oct 30 16:37:51.134981 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:51.134948 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:51.139059 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:51.139029 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:37:51.139304 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:51.139274 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:37:52.139625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:52.138789 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:37:52.139625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:52.138877 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:37:53.137627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:53.134910 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:37:53.137627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:53.134987 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:37:54.134912 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:54.134874 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:37:54.135431 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:54.134936 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:37:54.598466 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:37:54.598432 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:37:55.135938 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:55.135900 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:37:55.136479 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:55.136446 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:37:56.135181 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:56.135145 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 30 16:37:56.135347 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:56.135197 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 30 16:37:57.423627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:57.423577 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:57.424854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:57.424838 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:37:57.425464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:37:57.425446 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:38:04.220519 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.220488 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp"] Oct 30 16:38:04.225501 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.225479 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.270187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.270143 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp"] Oct 30 16:38:04.297021 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.296986 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm"] Oct 30 16:38:04.305875 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.305838 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.313946 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.313914 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm"] Oct 30 16:38:04.323981 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.323954 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr"] Oct 30 16:38:04.329759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.329728 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.351700 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.351677 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr"] Oct 30 16:38:04.415526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.415497 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkrmj\" (UniqueName: \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-kube-api-access-bkrmj\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.415768 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.415542 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.415768 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.415643 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.415768 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.415693 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.415900 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.415764 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.415900 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.415803 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.415900 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.415859 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.416050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.415904 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.416050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.415971 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.416050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.416000 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.416208 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.416077 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.416208 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.416106 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7295aef-3d34-43db-9a22-71f3b9a6b554-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.416208 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.416136 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.416208 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.416169 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.416208 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.416203 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.416453 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.416229 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.416453 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.416313 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.416453 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.416342 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m89p9\" (UniqueName: \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-kube-api-access-m89p9\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.517918 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.517890 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.518103 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.517924 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.518103 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.517969 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.518103 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.517989 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m89p9\" (UniqueName: \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-kube-api-access-m89p9\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.518103 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518015 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.518103 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518062 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkrmj\" (UniqueName: \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-kube-api-access-bkrmj\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.518103 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518086 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4256fb11-e481-46fb-9964-1706f25e8e71-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.518458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518177 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.518458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518213 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.518458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518237 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.518458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518264 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.518458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518296 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.518458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518397 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.518793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518546 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.518793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518638 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.518793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518667 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4256fb11-e481-46fb-9964-1706f25e8e71-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.518793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518717 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.518793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518729 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.518793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518736 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.518793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518641 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.518793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518748 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-certs\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.519105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518895 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.519105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518920 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.519105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518949 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5spqn\" (UniqueName: \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-kube-api-access-5spqn\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.519105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.518991 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.519105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519038 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.519105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519058 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.519105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519078 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7295aef-3d34-43db-9a22-71f3b9a6b554-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.519105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519094 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.519515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519121 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.519515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519138 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-credential-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.519515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519143 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.519515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519218 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.519515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519384 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-data\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.520424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519736 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-socket\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.520424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.519994 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.520424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.520228 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7295aef-3d34-43db-9a22-71f3b9a6b554-istiod-ca-cert\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.520868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.520846 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.521237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.521210 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-envoy\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.521561 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.521541 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.522023 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.522008 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-podinfo\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.538929 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.538909 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.542927 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.542903 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkrmj\" (UniqueName: \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-kube-api-access-bkrmj\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-94npm\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.543218 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.543201 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m89p9\" (UniqueName: \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-kube-api-access-m89p9\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.547264 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.547247 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-token\") pod \"e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.620682 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.620653 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.620814 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.620709 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.620814 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.620742 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.620814 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.620768 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.620991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.620885 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.620991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.620944 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4256fb11-e481-46fb-9964-1706f25e8e71-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.621085 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.621008 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-data\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.621085 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.621045 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.621085 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.621073 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4256fb11-e481-46fb-9964-1706f25e8e71-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.621241 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.621180 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5spqn\" (UniqueName: \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-kube-api-access-5spqn\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.621241 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.621201 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-credential-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.621404 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.621383 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-socket\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.621469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.621384 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-certs\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.621804 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.621778 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4256fb11-e481-46fb-9964-1706f25e8e71-istiod-ca-cert\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.623306 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.623284 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-envoy\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.623541 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.623521 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4256fb11-e481-46fb-9964-1706f25e8e71-istio-podinfo\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.633175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.633154 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:04.652494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.652468 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-istio-token\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.657990 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.657969 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5spqn\" (UniqueName: \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-kube-api-access-5spqn\") pod \"e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.839982 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.839906 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:04.943927 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.943895 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:04.955714 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.955694 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm"] Oct 30 16:38:04.959745 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:38:04.959716 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod649f1900_0d41_45c8_9cee_e1e6c7fd2610.slice/crio-8b08898b258acfb9fd68f7598d6114ff8ab62c9f03919da488f750389c948ed0 WatchSource:0}: Error finding container 8b08898b258acfb9fd68f7598d6114ff8ab62c9f03919da488f750389c948ed0: Status 404 returned error can't find the container with id 8b08898b258acfb9fd68f7598d6114ff8ab62c9f03919da488f750389c948ed0 Oct 30 16:38:04.962930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.962893 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:04.963017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.962970 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:04.963017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:04.962999 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.151486 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.151437 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp"] Oct 30 16:38:05.151968 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:38:05.151941 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7295aef_3d34_43db_9a22_71f3b9a6b554.slice/crio-cef8254979bcddbd2bbdfdb49880f5fa001c6125854e76f2dee90d043bc950ba WatchSource:0}: Error finding container cef8254979bcddbd2bbdfdb49880f5fa001c6125854e76f2dee90d043bc950ba: Status 404 returned error can't find the container with id cef8254979bcddbd2bbdfdb49880f5fa001c6125854e76f2dee90d043bc950ba Oct 30 16:38:05.154567 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.154539 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.154671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.154621 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.154671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.154651 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.261828 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.261790 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr"] Oct 30 16:38:05.262740 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:38:05.262709 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4256fb11_e481_46fb_9964_1706f25e8e71.slice/crio-4d5791244c0097b1abe7b6095dce0fe1855f9ba5425d90a2ca15b518c3a8580e WatchSource:0}: Error finding container 4d5791244c0097b1abe7b6095dce0fe1855f9ba5425d90a2ca15b518c3a8580e: Status 404 returned error can't find the container with id 4d5791244c0097b1abe7b6095dce0fe1855f9ba5425d90a2ca15b518c3a8580e Oct 30 16:38:05.266037 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.265994 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.266127 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.266086 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.266177 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.266125 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:38:05.500037 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.500000 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" event={"ID":"4256fb11-e481-46fb-9964-1706f25e8e71","Type":"ContainerStarted","Data":"cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1"} Oct 30 16:38:05.500207 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.500091 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" event={"ID":"4256fb11-e481-46fb-9964-1706f25e8e71","Type":"ContainerStarted","Data":"4d5791244c0097b1abe7b6095dce0fe1855f9ba5425d90a2ca15b518c3a8580e"} Oct 30 16:38:05.501763 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.501735 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" event={"ID":"e7295aef-3d34-43db-9a22-71f3b9a6b554","Type":"ContainerStarted","Data":"04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323"} Oct 30 16:38:05.501881 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.501787 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" event={"ID":"e7295aef-3d34-43db-9a22-71f3b9a6b554","Type":"ContainerStarted","Data":"cef8254979bcddbd2bbdfdb49880f5fa001c6125854e76f2dee90d043bc950ba"} Oct 30 16:38:05.503386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.503366 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" event={"ID":"649f1900-0d41-45c8-9cee-e1e6c7fd2610","Type":"ContainerStarted","Data":"98afc09d83e03534b36ab82bb94d70b5fd35667a5ef2d3817d7c57b9c089f611"} Oct 30 16:38:05.503477 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.503400 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" event={"ID":"649f1900-0d41-45c8-9cee-e1e6c7fd2610","Type":"ContainerStarted","Data":"8b08898b258acfb9fd68f7598d6114ff8ab62c9f03919da488f750389c948ed0"} Oct 30 16:38:05.543649 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.543586 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" podStartSLOduration=1.5435724199999998 podStartE2EDuration="1.54357242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:05.541153059 +0000 UTC m=+994.691434051" watchObservedRunningTime="2026-10-30 16:38:05.54357242 +0000 UTC m=+994.693853407" Oct 30 16:38:05.599201 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.599153 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" podStartSLOduration=1.5991347519999999 podStartE2EDuration="1.599134752s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:05.597537847 +0000 UTC m=+994.747818835" watchObservedRunningTime="2026-10-30 16:38:05.599134752 +0000 UTC m=+994.749415741" Oct 30 16:38:05.634334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.634309 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:05.641936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.641893 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podStartSLOduration=1.641881457 podStartE2EDuration="1.641881457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:38:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:38:05.638788434 +0000 UTC m=+994.789069423" watchObservedRunningTime="2026-10-30 16:38:05.641881457 +0000 UTC m=+994.792162444" Oct 30 16:38:05.840461 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.840369 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:05.944402 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:05.944367 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:06.639641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:06.639247 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:38:06.639641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:06.639315 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:38:06.842753 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:06.842718 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:38:06.843034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:06.843009 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 16:38:06.946683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:06.946576 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:38:06.946955 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:06.946925 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:38:07.642628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:07.633969 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:38:07.642628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:07.634028 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:38:07.842371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:07.841944 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:38:07.842371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:07.842006 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 16:38:07.948104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:07.948007 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:38:07.948358 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:07.948327 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:38:08.637630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:08.635027 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:38:08.637630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:08.635091 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:38:08.843749 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:08.843708 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:38:08.844347 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:08.844315 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 16:38:08.992300 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:08.991044 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:38:08.992300 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:08.991106 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:38:09.637188 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:09.637150 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.640687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:09.637450 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:38:09.854496 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:09.854460 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.855150 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:09.855120 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 16:38:09.951687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:09.946966 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:38:09.951687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:09.947031 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:38:10.649728 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:10.649677 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.650026 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:10.649983 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:38:10.844382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:10.844347 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.844691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:10.844663 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 16:38:10.951683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:10.947984 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:38:10.951683 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:10.948044 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:38:11.645905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:11.645872 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.646380 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:11.646341 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:38:11.841406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:11.841364 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.841630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:11.841447 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 16:38:11.951678 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:11.947007 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:38:11.951678 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:11.947095 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:38:12.638626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:12.634740 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.638626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:12.634799 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" Oct 30 16:38:12.840460 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:12.840424 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.840812 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:12.840765 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 30 16:38:12.944625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:12.944519 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 30 16:38:12.944625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:12.944583 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 30 16:38:13.784286 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:13.784248 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:13.784830 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:13.784689 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:13.785486 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:13.785458 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 16:38:13.910885 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:13.910854 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:13.911451 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:13.911426 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:13.912234 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:13.912218 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:38:14.985778 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:14.985748 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:38:14.986187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:14.985822 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:15.350336 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:15.350248 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:15.350700 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:15.350672 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:15.351527 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:15.351498 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:38:24.814153 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:38:24.814113 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:38:24.986628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:24.974505 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.044641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.018707 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.044641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.036758 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.122824 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.122703 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.124276 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.124242 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.124739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.124716 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:38:25.415684 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.406720 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.530631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.507736 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 16:38:25.530631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.508178 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 16:38:25.755628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.752705 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.764724 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.764680 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.806006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.805957 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:25.889037 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:25.888988 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.054687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.021723 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.054687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.022585 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:38:26.103628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.057362 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.160629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.141759 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:38:26.160629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.143121 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.160629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.144691 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 16:38:26.160629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.145021 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:26.160629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.145654 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:38:26.160629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.145712 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:38:26.160629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.145751 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:38:26.497313 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.497267 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:38:26.499283 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.499256 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:38:26.648530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.648492 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.651477 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.651455 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:38:26.652304 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.649001 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.653074 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.653057 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:38:26.806727 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.754144 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 16:38:26.847671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.841549 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:38:26.847888 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.765544 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:26.873655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:26.848070 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:38:27.027772 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.026646 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 16:38:27.142787 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.132062 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.147276 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.147238 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.150545 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.150520 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:38:27.169383 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.152212 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 16:38:27.179617 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.179560 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.195031 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.195001 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:38:27.227043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.227020 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:38:27.420105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.420024 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:38:27.500636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.500331 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.653657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.653593 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:27.671659 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.663470 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:38:27.671659 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.664524 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:38:27.693059 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.693030 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:38:27.702875 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.702846 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:38:27.706985 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.706960 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:38:27.758081 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:27.758053 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:38:38.040940 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:38.039722 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.166746 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:38.166701 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.175139 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:38.175099 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.220981 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:38.220937 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:38.497687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:38.496741 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.135300 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:39.135255 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:38:39.246246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:39.245329 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:38:39.250553 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:39.248213 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:38:39.339284 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:39.339253 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:38:54.598988 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:38:54.598946 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:38:59.944818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:38:59.944739 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": context deadline exceeded" Oct 30 16:39:00.031758 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:00.031710 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.160333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:00.147692 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 16:39:00.160333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:00.159435 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:39:00.200722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:00.188844 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.247787 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:00.247735 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.345972 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:00.345922 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.436248 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:00.436202 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.663750 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:00.663689 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:00.721767 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:00.721716 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.032285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.032225 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 16:39:01.035626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.033766 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:39:01.176902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.176848 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.187777 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.187723 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.334692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.198995 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:39:01.334692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.299857 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:39:01.334692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.271718 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.334692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.200933 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.334692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.301228 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:39:01.334692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.301278 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:39:01.414632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.412404 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.414632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.412942 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:39:01.490202 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.471720 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.490202 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.482754 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:39:01.792538 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.557864 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:39:01.793500 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.580871 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:39:01.793836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.793816 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:39:01.793998 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.793984 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:39:01.879089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.665284 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.879089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.722441 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:01.879089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.796785 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:39:01.879089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.796840 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:39:01.917650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.917623 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:39:01.921471 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:01.921446 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:39:02.058770 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:02.048241 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:39:02.058770 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:02.056057 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:39:02.061669 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:02.061378 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:39:02.087419 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:02.086488 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:39:14.052629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:14.051697 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:14.057679 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:14.057644 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 16:39:14.172629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:14.171721 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 16:39:14.660627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:14.659738 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:14.827790 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:14.827742 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:39:14.827975 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:14.827824 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:24.602808 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:39:24.602763 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:39:52.611132 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:52.611080 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 16:39:52.747629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:52.746756 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:52.926047 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:52.925795 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:52.927471 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:52.927441 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.053628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:53.052706 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.059633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:53.057688 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.070633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:53.070445 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.094063 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:53.094009 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.511632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:53.507726 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:39:53.615785 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:53.615726 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.617469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:53.617446 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:39:53.619175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:53.619139 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.970704 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:53.942801 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:53.970704 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:53.943565 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:39:54.119638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.113305 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:54.119638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.113756 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 16:39:54.119638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.114062 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 16:39:54.119638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.115918 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:39:54.119638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.116004 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:39:54.119638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.116066 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:39:54.132667 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.121928 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 16:39:54.132667 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.122641 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:39:54.184807 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.184757 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": context deadline exceeded" Oct 30 16:39:54.185785 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.185763 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:39:54.462322 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.462293 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:39:54.524294 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.524245 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:54.525826 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.525802 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:39:54.630819 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.630773 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:39:54.962628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.912843 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:39:54.962628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.945010 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:39:54.986628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:54.965902 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:39:55.076971 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:55.076922 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 16:39:55.146754 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:55.145192 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:39:55.146754 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:55.145273 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:39:55.146754 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:55.145306 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:39:55.146754 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:55.145336 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:39:55.151816 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:39:55.151778 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:39:57.300359 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.300319 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb"] Oct 30 16:39:57.303803 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.303776 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.328747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.328710 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3e2f96\"/\"distinct-w0-3e2f96-kserve-self-signed-certs\"" Oct 30 16:39:57.332334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.332308 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb"] Oct 30 16:39:57.351445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.351416 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzt6w\" (UniqueName: \"kubernetes.io/projected/6927cc23-033e-4bad-9a12-573afcf72cf0-kube-api-access-wzt6w\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.351630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.351550 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-model-cache\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.351630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.351583 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-home\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.351758 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.351711 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-dshm\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.351817 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.351803 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6927cc23-033e-4bad-9a12-573afcf72cf0-tls-certs\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.351869 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.351836 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-tmp-dir\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.453409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.453373 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-model-cache\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.453639 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.453420 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-home\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.453639 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.453465 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-dshm\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.453639 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.453530 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6927cc23-033e-4bad-9a12-573afcf72cf0-tls-certs\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.453639 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.453571 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-tmp-dir\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.453871 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.453663 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzt6w\" (UniqueName: \"kubernetes.io/projected/6927cc23-033e-4bad-9a12-573afcf72cf0-kube-api-access-wzt6w\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.453930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.453892 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-model-cache\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.454055 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.454032 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-tmp-dir\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.454219 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.454199 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-home\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.456042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.455998 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-dshm\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.456752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.456728 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6927cc23-033e-4bad-9a12-573afcf72cf0-tls-certs\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.491666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.491632 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzt6w\" (UniqueName: \"kubernetes.io/projected/6927cc23-033e-4bad-9a12-573afcf72cf0-kube-api-access-wzt6w\") pod \"distinct-w0-3e2f96-kserve-6465bc6846-nvrsb\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.617892 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.617801 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:57.934648 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:57.934577 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb"] Oct 30 16:39:57.935778 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:39:57.935749 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6927cc23_033e_4bad_9a12_573afcf72cf0.slice/crio-4828f85b06bb5166d400ac227bf5e915ef72976b2bef897ea3eac67081943719 WatchSource:0}: Error finding container 4828f85b06bb5166d400ac227bf5e915ef72976b2bef897ea3eac67081943719: Status 404 returned error can't find the container with id 4828f85b06bb5166d400ac227bf5e915ef72976b2bef897ea3eac67081943719 Oct 30 16:39:58.471873 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:58.471838 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" event={"ID":"6927cc23-033e-4bad-9a12-573afcf72cf0","Type":"ContainerStarted","Data":"d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337"} Oct 30 16:39:58.472260 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:58.471890 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" event={"ID":"6927cc23-033e-4bad-9a12-573afcf72cf0","Type":"ContainerStarted","Data":"4828f85b06bb5166d400ac227bf5e915ef72976b2bef897ea3eac67081943719"} Oct 30 16:39:58.511255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:58.511207 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podStartSLOduration=1.511190847 podStartE2EDuration="1.511190847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:39:58.509147349 +0000 UTC m=+1107.659428328" watchObservedRunningTime="2026-10-30 16:39:58.511190847 +0000 UTC m=+1107.661471833" Oct 30 16:39:58.618584 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:58.618552 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:39:59.173091 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.173055 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54"] Oct 30 16:39:59.177116 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.177094 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.199386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.199361 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 16:39:59.199386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.199363 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:39:59.203160 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.203132 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54"] Oct 30 16:39:59.285170 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.285140 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l58q6\" (UniqueName: \"kubernetes.io/projected/750c619f-fc91-4814-99a9-66b3015350bb-kube-api-access-l58q6\") pod \"maas-api-revoke-sub-e2a1fd15-jxw54\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.285383 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.285200 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a1fd15-jxw54\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.285383 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.285287 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a1fd15-jxw54\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.387057 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.387023 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l58q6\" (UniqueName: \"kubernetes.io/projected/750c619f-fc91-4814-99a9-66b3015350bb-kube-api-access-l58q6\") pod \"maas-api-revoke-sub-e2a1fd15-jxw54\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.387233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.387082 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a1fd15-jxw54\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.387233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.387108 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a1fd15-jxw54\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.387813 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.387788 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2a1fd15-jxw54\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.389663 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.389641 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2a1fd15-jxw54\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.409116 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.409091 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l58q6\" (UniqueName: \"kubernetes.io/projected/750c619f-fc91-4814-99a9-66b3015350bb-kube-api-access-l58q6\") pod \"maas-api-revoke-sub-e2a1fd15-jxw54\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.491285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.491252 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:39:59.802553 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:39:59.802521 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54"] Oct 30 16:39:59.803033 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:39:59.803011 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750c619f_fc91_4814_99a9_66b3015350bb.slice/crio-cff05959508a6dadb4fb60cec1769628f404aa4721b8d88d15e3dc488479e79d WatchSource:0}: Error finding container cff05959508a6dadb4fb60cec1769628f404aa4721b8d88d15e3dc488479e79d: Status 404 returned error can't find the container with id cff05959508a6dadb4fb60cec1769628f404aa4721b8d88d15e3dc488479e79d Oct 30 16:40:00.489032 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:00.488987 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" event={"ID":"750c619f-fc91-4814-99a9-66b3015350bb","Type":"ContainerStarted","Data":"a0a9233e0e95ea67f206aa606788b2ec941dfa7231af6a575a21933333a1e633"} Oct 30 16:40:00.489222 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:00.489042 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" event={"ID":"750c619f-fc91-4814-99a9-66b3015350bb","Type":"ContainerStarted","Data":"cff05959508a6dadb4fb60cec1769628f404aa4721b8d88d15e3dc488479e79d"} Oct 30 16:40:00.529343 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:00.529285 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" podStartSLOduration=1.529270012 podStartE2EDuration="1.529270012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:39:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:00.526552249 +0000 UTC m=+1109.676833240" watchObservedRunningTime="2026-10-30 16:40:00.529270012 +0000 UTC m=+1109.679551335" Oct 30 16:40:01.497706 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:01.497672 6400 generic.go:356] "Generic (PLEG): container finished" podID="750c619f-fc91-4814-99a9-66b3015350bb" containerID="a0a9233e0e95ea67f206aa606788b2ec941dfa7231af6a575a21933333a1e633" exitCode=0 Oct 30 16:40:01.497887 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:01.497753 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" event={"ID":"750c619f-fc91-4814-99a9-66b3015350bb","Type":"ContainerDied","Data":"a0a9233e0e95ea67f206aa606788b2ec941dfa7231af6a575a21933333a1e633"} Oct 30 16:40:02.682339 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:02.682310 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:40:02.745755 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:02.745725 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-openshift-ca\") pod \"750c619f-fc91-4814-99a9-66b3015350bb\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " Oct 30 16:40:02.745951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:02.745853 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-xks-ca\") pod \"750c619f-fc91-4814-99a9-66b3015350bb\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " Oct 30 16:40:02.746021 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:02.745969 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750c619f-fc91-4814-99a9-66b3015350bb-kube-api-access-l58q6\" (UniqueName: \"kubernetes.io/projected/750c619f-fc91-4814-99a9-66b3015350bb-kube-api-access-l58q6\") pod \"750c619f-fc91-4814-99a9-66b3015350bb\" (UID: \"750c619f-fc91-4814-99a9-66b3015350bb\") " Oct 30 16:40:02.746154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:02.746131 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-openshift-ca" pod "750c619f-fc91-4814-99a9-66b3015350bb" (UID: "750c619f-fc91-4814-99a9-66b3015350bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:40:02.746629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:02.746553 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:40:02.748225 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:02.748203 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750c619f-fc91-4814-99a9-66b3015350bb-kube-api-access-l58q6" pod "750c619f-fc91-4814-99a9-66b3015350bb" (UID: "750c619f-fc91-4814-99a9-66b3015350bb"). InnerVolumeSpecName "kube-api-access-l58q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:40:02.748352 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:02.748269 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-xks-ca" pod "750c619f-fc91-4814-99a9-66b3015350bb" (UID: "750c619f-fc91-4814-99a9-66b3015350bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:40:02.847655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:02.847624 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750c619f-fc91-4814-99a9-66b3015350bb-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:40:02.847655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:02.847652 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l58q6\" (UniqueName: \"kubernetes.io/projected/750c619f-fc91-4814-99a9-66b3015350bb-kube-api-access-l58q6\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:40:03.513135 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:03.513104 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" Oct 30 16:40:03.513294 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:03.513143 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54" event={"ID":"750c619f-fc91-4814-99a9-66b3015350bb","Type":"ContainerDied","Data":"cff05959508a6dadb4fb60cec1769628f404aa4721b8d88d15e3dc488479e79d"} Oct 30 16:40:03.513294 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:03.513196 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cff05959508a6dadb4fb60cec1769628f404aa4721b8d88d15e3dc488479e79d" Oct 30 16:40:04.356860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.356828 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl"] Oct 30 16:40:04.357464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.357447 6400 cpu_manager.go:414] "containerMap: removing container" podUID="750c619f-fc91-4814-99a9-66b3015350bb" containerName="revoke-keys" Oct 30 16:40:04.357464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.357465 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750c619f-fc91-4814-99a9-66b3015350bb" containerName="revoke-keys" Oct 30 16:40:04.357555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.357548 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="750c619f-fc91-4814-99a9-66b3015350bb" containerName="revoke-keys" Oct 30 16:40:04.360120 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.360105 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.388204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.388159 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl"] Oct 30 16:40:04.468845 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.468814 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.469010 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.468973 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.469010 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.468997 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.469094 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.469080 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.469141 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.469106 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.469195 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.469159 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmr4h\" (UniqueName: \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-kube-api-access-gmr4h\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.469248 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.469210 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.469299 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.469250 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.469347 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.469333 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571005 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.570967 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571005 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571006 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571053 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571075 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571095 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmr4h\" (UniqueName: \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-kube-api-access-gmr4h\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571132 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571163 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571236 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571333 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571405 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571447 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571778 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571753 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.571889 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.571864 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.572162 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.572140 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.573860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.573837 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.573943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.573934 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.592203 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.592180 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.592449 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.592427 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmr4h\" (UniqueName: \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-kube-api-access-gmr4h\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:04.676540 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:04.676462 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:05.001378 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:05.001272 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl"] Oct 30 16:40:05.005397 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:05.005340 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:05.005515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:05.005484 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:05.005615 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:05.005581 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:05.536635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:05.535872 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" event={"ID":"e4b1aabb-56f0-4e96-a71b-a272a57b99a3","Type":"ContainerStarted","Data":"f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978"} Oct 30 16:40:05.536635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:05.535945 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" event={"ID":"e4b1aabb-56f0-4e96-a71b-a272a57b99a3","Type":"ContainerStarted","Data":"c95d98c635cbe3da20aee95f4678d9d3074db95819f96d2103b0f9dbf827e022"} Oct 30 16:40:05.591745 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:05.591678 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" podStartSLOduration=1.591658642 podStartE2EDuration="1.591658642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:05.588902954 +0000 UTC m=+1114.739183941" watchObservedRunningTime="2026-10-30 16:40:05.591658642 +0000 UTC m=+1114.741939632" Oct 30 16:40:05.677580 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:05.677547 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:06.680474 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:06.680439 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:40:06.681138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:06.681106 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:40:07.623073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:07.623041 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:40:07.692635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:07.686946 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:40:07.692635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:07.687009 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:40:08.109645 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:08.109020 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:40:08.678026 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:08.677985 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:40:08.678357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:08.678315 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:40:09.097877 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:09.096243 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:40:09.677954 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:09.677923 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:40:09.678158 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:09.677980 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:40:10.677079 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:10.677046 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:40:10.677515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:10.677105 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:40:11.677200 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:11.677164 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 30 16:40:11.677615 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:11.677219 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 30 16:40:12.978420 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:12.978384 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:12.978920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:12.978901 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:12.980269 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:12.980248 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:40:18.914268 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:18.914237 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh"] Oct 30 16:40:18.918592 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:18.918571 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:18.950739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:18.950699 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh"] Oct 30 16:40:19.022402 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.022314 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2"] Oct 30 16:40:19.027104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.027073 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.045657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.045620 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-3e2f96-openshift-default-dockercfg-v5cl2\"" Oct 30 16:40:19.052680 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.052589 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2"] Oct 30 16:40:19.097655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.097593 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.097863 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.097666 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.097863 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.097708 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.098004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.097878 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.098065 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.098011 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db06720-8b98-4659-be16-f1a3429b0a06-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.098065 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.098029 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6sr5\" (UniqueName: \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-kube-api-access-v6sr5\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.098172 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.098143 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.098172 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.098168 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.098350 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.098194 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db06720-8b98-4659-be16-f1a3429b0a06-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.199226 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199128 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.199226 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199199 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.199431 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199286 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.199431 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199319 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.199431 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199353 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.199551 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199500 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db06720-8b98-4659-be16-f1a3429b0a06-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.199551 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199530 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6sr5\" (UniqueName: \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-kube-api-access-v6sr5\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.199654 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199559 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.199654 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199587 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.199654 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199635 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.199797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199699 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.199797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199728 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.199797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199750 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.199797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199780 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db06720-8b98-4659-be16-f1a3429b0a06-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.200014 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199832 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.200014 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199903 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.200014 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.199979 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp9ms\" (UniqueName: \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-kube-api-access-lp9ms\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.200150 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.200031 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.200150 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.200034 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.200229 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.200203 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.200266 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.200241 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.200575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.200553 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.200668 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.200654 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db06720-8b98-4659-be16-f1a3429b0a06-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.202162 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.202138 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db06720-8b98-4659-be16-f1a3429b0a06-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.202523 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.202508 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.221499 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.221477 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6sr5\" (UniqueName: \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-kube-api-access-v6sr5\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.221919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.221902 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.237319 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.237300 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304015 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304058 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp9ms\" (UniqueName: \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-kube-api-access-lp9ms\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304094 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304225 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304289 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304318 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304446 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304457 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304567 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304701 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.304842 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.305395 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.306300 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.307164 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.312424 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.313686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.312635 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.329329 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.329285 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp9ms\" (UniqueName: \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-kube-api-access-lp9ms\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.331452 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.331019 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.351307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.351260 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:19.559031 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.559006 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh"] Oct 30 16:40:19.560286 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:40:19.560263 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9db06720_8b98_4659_be16_f1a3429b0a06.slice/crio-0728d2c8f8f03cb51bde9079c77eee3805ab3627bd2f61472ac0a317b01ef21b WatchSource:0}: Error finding container 0728d2c8f8f03cb51bde9079c77eee3805ab3627bd2f61472ac0a317b01ef21b: Status 404 returned error can't find the container with id 0728d2c8f8f03cb51bde9079c77eee3805ab3627bd2f61472ac0a317b01ef21b Oct 30 16:40:19.562891 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.562858 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:19.562966 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.562930 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:19.563017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.562965 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:19.685944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.685913 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2"] Oct 30 16:40:19.689084 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:40:19.689049 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61744ea0_6fdc_4a3c_bcba_f93f6aa69d89.slice/crio-09ddbd4c1e3a58304ab2b1ed5e5812fece24cc726cf034ead9ad2fee45589182 WatchSource:0}: Error finding container 09ddbd4c1e3a58304ab2b1ed5e5812fece24cc726cf034ead9ad2fee45589182: Status 404 returned error can't find the container with id 09ddbd4c1e3a58304ab2b1ed5e5812fece24cc726cf034ead9ad2fee45589182 Oct 30 16:40:19.692140 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.692103 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:19.692242 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.692185 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:19.692242 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.692229 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:40:19.743174 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.743120 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" event={"ID":"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89","Type":"ContainerStarted","Data":"09ddbd4c1e3a58304ab2b1ed5e5812fece24cc726cf034ead9ad2fee45589182"} Oct 30 16:40:19.745333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.745255 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" event={"ID":"9db06720-8b98-4659-be16-f1a3429b0a06","Type":"ContainerStarted","Data":"b72c805285238641028db505aa3fe2aa9edbf70607c90e19a41d215305af67e8"} Oct 30 16:40:19.745333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.745305 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" event={"ID":"9db06720-8b98-4659-be16-f1a3429b0a06","Type":"ContainerStarted","Data":"0728d2c8f8f03cb51bde9079c77eee3805ab3627bd2f61472ac0a317b01ef21b"} Oct 30 16:40:19.789334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:19.789268 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" podStartSLOduration=1.789249713 podStartE2EDuration="1.789249713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:19.786256747 +0000 UTC m=+1128.936537736" watchObservedRunningTime="2026-10-30 16:40:19.789249713 +0000 UTC m=+1128.939530704" Oct 30 16:40:20.237806 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:20.237762 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:20.240518 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:20.240481 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:40:20.240788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:20.240754 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:40:20.797861 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:20.797818 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" event={"ID":"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89","Type":"ContainerStarted","Data":"6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d"} Oct 30 16:40:20.877007 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:20.876947 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" podStartSLOduration=2.87692957 podStartE2EDuration="2.87692957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:20.863067843 +0000 UTC m=+1130.013348834" watchObservedRunningTime="2026-10-30 16:40:20.87692957 +0000 UTC m=+1130.027210582" Oct 30 16:40:21.238096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:21.238055 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:40:21.238777 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:21.238739 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:40:21.351989 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:21.351958 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:21.355736 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:21.355705 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:40:21.355968 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:21.355936 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 16:40:22.239007 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:22.238970 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:40:22.241112 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:22.241078 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:40:22.352070 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:22.352032 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:40:22.352388 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:22.352360 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 16:40:23.239093 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:23.239059 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:40:23.239788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:23.239751 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:40:23.352023 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:23.351983 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:40:23.352329 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:23.352301 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 16:40:24.248629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:24.241015 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:40:24.248629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:24.241076 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:40:24.353628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:24.352010 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:40:24.353628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:24.352071 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 16:40:24.598493 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:40:24.598410 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:40:25.246629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:25.246029 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 30 16:40:25.246629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:25.246110 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 30 16:40:25.360092 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:25.360050 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:40:25.360761 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:25.360721 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 16:40:26.351711 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:26.351670 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body= Oct 30 16:40:26.351931 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:26.351753 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" Oct 30 16:40:26.491493 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:26.491365 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:26.492650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:26.492559 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:26.496924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:26.494398 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:40:27.366627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:27.366261 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:27.369659 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:27.367764 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:27.369659 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:27.369235 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 16:40:28.680625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:28.679715 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 16:40:28.871778 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:28.871719 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:28.931800 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:28.931696 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:29.029577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:29.029526 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:29.190911 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:29.190781 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:36.575184 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.575148 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp"] Oct 30 16:40:36.583006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.582980 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.608541 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.608512 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"openshift-service-ca.crt\"" Oct 30 16:40:36.608952 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.608935 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"kube-root-ca.crt\"" Oct 30 16:40:36.611644 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.611625 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"default-dockercfg-xznkg\"" Oct 30 16:40:36.611792 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.611724 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-d693d0\"/\"distinct-w6-d693d0-kserve-self-signed-certs\"" Oct 30 16:40:36.612276 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.612256 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp"] Oct 30 16:40:36.669326 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.669286 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-tmp-dir\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.669326 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.669330 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-dshm\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.669583 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.669536 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-model-cache\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.669583 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.669566 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l4tp\" (UniqueName: \"kubernetes.io/projected/8fc34352-0241-44b1-96fe-ddccd74ac59a-kube-api-access-8l4tp\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.669728 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.669705 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-home\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.669787 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.669740 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc34352-0241-44b1-96fe-ddccd74ac59a-tls-certs\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.775876 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.772972 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-home\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.775876 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.773055 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc34352-0241-44b1-96fe-ddccd74ac59a-tls-certs\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.775876 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.773305 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-tmp-dir\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.775876 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.773371 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-dshm\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.775876 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.773580 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-model-cache\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.775876 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.773630 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l4tp\" (UniqueName: \"kubernetes.io/projected/8fc34352-0241-44b1-96fe-ddccd74ac59a-kube-api-access-8l4tp\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.775876 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.774464 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-home\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.780622 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.778416 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-tmp-dir\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.780622 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.778697 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-model-cache\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.784475 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.783869 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc34352-0241-44b1-96fe-ddccd74ac59a-tls-certs\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.784475 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.784062 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-dshm\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.803304 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.803271 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l4tp\" (UniqueName: \"kubernetes.io/projected/8fc34352-0241-44b1-96fe-ddccd74ac59a-kube-api-access-8l4tp\") pod \"distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:36.900965 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:36.900826 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:37.242653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:37.242576 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp"] Oct 30 16:40:37.243419 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:40:37.243395 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8fc34352_0241_44b1_96fe_ddccd74ac59a.slice/crio-ad74e0369d77598ca122838fe07a422de26d319d50c9727f8d1b85bddbb3fdcc WatchSource:0}: Error finding container ad74e0369d77598ca122838fe07a422de26d319d50c9727f8d1b85bddbb3fdcc: Status 404 returned error can't find the container with id ad74e0369d77598ca122838fe07a422de26d319d50c9727f8d1b85bddbb3fdcc Oct 30 16:40:38.118382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:38.118345 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" event={"ID":"8fc34352-0241-44b1-96fe-ddccd74ac59a","Type":"ContainerStarted","Data":"47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8"} Oct 30 16:40:38.118814 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:38.118391 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" event={"ID":"8fc34352-0241-44b1-96fe-ddccd74ac59a","Type":"ContainerStarted","Data":"ad74e0369d77598ca122838fe07a422de26d319d50c9727f8d1b85bddbb3fdcc"} Oct 30 16:40:38.153455 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:38.153402 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podStartSLOduration=2.153387504 podStartE2EDuration="2.153387504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:38.152674333 +0000 UTC m=+1147.302955320" watchObservedRunningTime="2026-10-30 16:40:38.153387504 +0000 UTC m=+1147.303668492" Oct 30 16:40:38.901195 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:38.901161 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:41.592880 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:41.592158 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 16:40:41.917686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:41.916693 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:41.920671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:41.917912 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:41.959120 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:41.959057 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.186886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:42.186756 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:40:42.385799 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:42.385740 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.544629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:42.544571 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:42.552758 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:42.552717 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.594794 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:42.594736 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.596795 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:42.595668 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:40:42.804099 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:42.803996 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.919772 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:42.919724 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:42.945626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:42.920865 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:40:43.023006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.012708 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.023006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.012816 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 16:40:43.023006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.013247 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:40:43.049630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.049161 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.126681 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.074301 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.126681 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.075755 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:40:43.126681 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.121806 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:40:43.188500 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.187231 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:43.383890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.383807 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:40:43.384161 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.384145 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:40:43.384309 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.384296 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:40:43.384441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.384430 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:40:43.384563 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.384553 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:40:43.384674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:43.384663 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:40:44.469367 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.469336 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq"] Oct 30 16:40:44.474854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.474825 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.504563 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.504534 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq"] Oct 30 16:40:44.506968 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.506939 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-3e2f96\"/\"distinct-2-w0-3e2f96-kserve-self-signed-certs\"" Oct 30 16:40:44.612126 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.612078 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-model-cache\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.612297 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.612192 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-home\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.612297 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.612253 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tmp-dir\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.612375 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.612307 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-dshm\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.612375 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.612365 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tls-certs\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.612506 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.612488 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmm85\" (UniqueName: \"kubernetes.io/projected/6ae453e0-0c2c-4080-bef3-c2119549a4ef-kube-api-access-fmm85\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.713662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.713625 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmm85\" (UniqueName: \"kubernetes.io/projected/6ae453e0-0c2c-4080-bef3-c2119549a4ef-kube-api-access-fmm85\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.713905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.713884 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-model-cache\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.713983 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.713960 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-home\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.714045 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.713986 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tmp-dir\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.714094 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.714070 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-dshm\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.714142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.714100 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tls-certs\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.714587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.714373 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-model-cache\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.714587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.714552 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tmp-dir\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.714810 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.714636 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-home\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.716920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.716891 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-dshm\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.717482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.717461 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tls-certs\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.737553 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.737448 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmm85\" (UniqueName: \"kubernetes.io/projected/6ae453e0-0c2c-4080-bef3-c2119549a4ef-kube-api-access-fmm85\") pod \"distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:44.808623 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:44.804753 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:45.131393 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:45.131362 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq"] Oct 30 16:40:45.132595 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:40:45.132570 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ae453e0_0c2c_4080_bef3_c2119549a4ef.slice/crio-55d4f3f06591b412d9379c61b6c49de1ae3ec7d8d48bdaf7137c1f951929ccd1 WatchSource:0}: Error finding container 55d4f3f06591b412d9379c61b6c49de1ae3ec7d8d48bdaf7137c1f951929ccd1: Status 404 returned error can't find the container with id 55d4f3f06591b412d9379c61b6c49de1ae3ec7d8d48bdaf7137c1f951929ccd1 Oct 30 16:40:45.374988 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:45.374948 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" event={"ID":"6ae453e0-0c2c-4080-bef3-c2119549a4ef","Type":"ContainerStarted","Data":"b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f"} Oct 30 16:40:45.374988 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:45.374989 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" event={"ID":"6ae453e0-0c2c-4080-bef3-c2119549a4ef","Type":"ContainerStarted","Data":"55d4f3f06591b412d9379c61b6c49de1ae3ec7d8d48bdaf7137c1f951929ccd1"} Oct 30 16:40:45.412811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:45.412711 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podStartSLOduration=1.4126937960000001 podStartE2EDuration="1.412693796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:40:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:40:45.411579768 +0000 UTC m=+1154.561860757" watchObservedRunningTime="2026-10-30 16:40:45.412693796 +0000 UTC m=+1154.562974784" Oct 30 16:40:45.805697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:45.805670 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:46.901613 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:46.901568 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:46.914874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:46.914847 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:47.399820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:47.399793 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:40:54.604367 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:40:54.604326 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:40:54.806341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:54.806306 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:54.818897 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:54.818872 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:55.481411 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:55.481369 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:57.081869 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:57.070968 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 16:40:57.156821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:57.156766 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.220005 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:57.219956 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.369682 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:57.369458 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.384625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:57.374673 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 16:40:57.455748 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:57.455696 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:57.776626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:57.775708 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded" Oct 30 16:40:57.903629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:57.902695 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.326041 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.325992 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:40:58.326777 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.326718 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:40:58.350172 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.350131 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.364742 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.364694 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.376751 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.376706 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.392465 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.387178 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:40:58.418498 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.409353 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": context deadline exceeded" Oct 30 16:40:58.476727 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.467814 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.478202 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.478164 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:40:58.485685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.480912 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:58.500623 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.500560 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:58.505703 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.505666 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.63:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.534147 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.505539 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:40:58.547629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.537832 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.751651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.631162 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:40:58.753362 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.647674 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:58.753886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.725746 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:40:58.846768 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:58.844722 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.215993 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:59.215907 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:40:59.428626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:59.406252 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:40:59.513996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:59.513967 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:40:59.685331 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:40:59.685285 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:04.011512 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.011478 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d"] Oct 30 16:41:04.022854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.022826 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.044633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.040134 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-750dcb\"/\"distinct-2-w2-750dcb-kserve-self-signed-certs\"" Oct 30 16:41:04.053844 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.053813 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d"] Oct 30 16:41:04.083316 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.083279 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws"] Oct 30 16:41:04.088744 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.088713 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.104265 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.101418 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-tmp-dir\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.104265 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.101512 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79cf7f9-6003-44e3-8064-eca14f082220-tls-certs\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.104265 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.101589 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jlnbs\" (UniqueName: \"kubernetes.io/projected/a79cf7f9-6003-44e3-8064-eca14f082220-kube-api-access-jlnbs\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.104265 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.101649 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-dshm\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.104265 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.101763 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-home\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.104265 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.101843 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-model-cache\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.111845 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.111819 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws"] Oct 30 16:41:04.182821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.182783 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw"] Oct 30 16:41:04.187124 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.187097 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.210957 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.210923 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw"] Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278239 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-home\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278325 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-model-cache\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278365 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a965f27d-01a1-4d25-9331-91fe73051186-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278403 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278427 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278486 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-tmp-dir\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278526 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278617 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278645 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-dshm\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278687 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79cf7f9-6003-44e3-8064-eca14f082220-tls-certs\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278712 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a965f27d-01a1-4d25-9331-91fe73051186-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278774 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jlnbs\" (UniqueName: \"kubernetes.io/projected/a79cf7f9-6003-44e3-8064-eca14f082220-kube-api-access-jlnbs\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.278920 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.279532 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-tmp-dir\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.280617 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-home\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.280877 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-model-cache\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.279123 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.291098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.281562 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q4t5\" (UniqueName: \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-kube-api-access-2q4t5\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.321445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.321371 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jlnbs\" (UniqueName: \"kubernetes.io/projected/a79cf7f9-6003-44e3-8064-eca14f082220-kube-api-access-jlnbs\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.344622 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.340572 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79cf7f9-6003-44e3-8064-eca14f082220-tls-certs\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.344622 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.340877 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-dshm\") pod \"distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.362490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.362208 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:04.382714 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.382677 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.382881 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.382853 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.382975 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.382891 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.382975 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.382919 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q4t5\" (UniqueName: \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-kube-api-access-2q4t5\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.383086 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.382995 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.383139 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383098 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.383187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383136 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25mzr\" (UniqueName: \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-kube-api-access-25mzr\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.383187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383170 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a965f27d-01a1-4d25-9331-91fe73051186-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.383296 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383206 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.383296 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383229 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.383296 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383286 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.383524 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383341 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.383524 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383375 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.383524 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383431 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a965f27d-01a1-4d25-9331-91fe73051186-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.383524 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383457 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.383524 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383511 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.383769 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383573 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.383769 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.383657 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.384943 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.384113 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-certs\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.387685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.385314 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.387685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.385591 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-credential-socket\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.387685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.385994 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a965f27d-01a1-4d25-9331-91fe73051186-istiod-ca-cert\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.387685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.386048 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-data\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.421028 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.420981 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-envoy\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.421418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.421388 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-istio-token\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.426270 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.425877 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q4t5\" (UniqueName: \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-kube-api-access-2q4t5\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.426834 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.426808 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a965f27d-01a1-4d25-9331-91fe73051186-istio-podinfo\") pod \"e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.488049 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.488011 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.488203 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.488080 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.488203 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.488169 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.488325 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.488210 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25mzr\" (UniqueName: \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-kube-api-access-25mzr\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.488325 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.488310 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.488490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.488364 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.488490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.488407 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.488490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.488465 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.488759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.488566 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.489368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.489340 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.491753 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.491671 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.492178 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.492154 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.492426 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.492405 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.492901 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.492881 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.493078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.493057 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.509893 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.509862 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.512661 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.512618 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25mzr\" (UniqueName: \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-kube-api-access-25mzr\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.521436 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.521407 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.723152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.723077 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:04.733802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.733779 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d"] Oct 30 16:41:04.735758 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:41:04.735500 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda79cf7f9_6003_44e3_8064_eca14f082220.slice/crio-9ea7cac6f21c1ecdb975791105e196f06606cf4c5dbf28ee6c9807ecc6fad917 WatchSource:0}: Error finding container 9ea7cac6f21c1ecdb975791105e196f06606cf4c5dbf28ee6c9807ecc6fad917: Status 404 returned error can't find the container with id 9ea7cac6f21c1ecdb975791105e196f06606cf4c5dbf28ee6c9807ecc6fad917 Oct 30 16:41:04.806078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.806051 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:04.883132 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.882069 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" event={"ID":"a79cf7f9-6003-44e3-8064-eca14f082220","Type":"ContainerStarted","Data":"8ad455e79b95b7edde2224c02fee74bf5f39d05de1e48f84497073f824e56a29"} Oct 30 16:41:04.883132 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.882127 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" event={"ID":"a79cf7f9-6003-44e3-8064-eca14f082220","Type":"ContainerStarted","Data":"9ea7cac6f21c1ecdb975791105e196f06606cf4c5dbf28ee6c9807ecc6fad917"} Oct 30 16:41:04.926261 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:04.926204 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" podStartSLOduration=1.926184203 podStartE2EDuration="1.926184203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:04.920772859 +0000 UTC m=+1174.071053847" watchObservedRunningTime="2026-10-30 16:41:04.926184203 +0000 UTC m=+1174.076465194" Oct 30 16:41:05.046296 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.046262 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws"] Oct 30 16:41:05.048633 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:41:05.048581 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda965f27d_01a1_4d25_9331_91fe73051186.slice/crio-2f49f394b1c48574f829fb35590debf9bab27d361e26d0af38a774009c794eef WatchSource:0}: Error finding container 2f49f394b1c48574f829fb35590debf9bab27d361e26d0af38a774009c794eef: Status 404 returned error can't find the container with id 2f49f394b1c48574f829fb35590debf9bab27d361e26d0af38a774009c794eef Oct 30 16:41:05.052345 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.052309 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:05.053561 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.052394 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:05.053561 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.052434 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:05.184594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.184555 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw"] Oct 30 16:41:05.208859 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.195352 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:05.208859 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.195440 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:05.208859 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.195486 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:05.363771 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.363663 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:05.890821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.890655 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" event={"ID":"a965f27d-01a1-4d25-9331-91fe73051186","Type":"ContainerStarted","Data":"3086068694d0be0ba4b5014c15bdcf29e668b00559ebcd5ee1a0789ea57d7406"} Oct 30 16:41:05.890821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.890714 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" event={"ID":"a965f27d-01a1-4d25-9331-91fe73051186","Type":"ContainerStarted","Data":"2f49f394b1c48574f829fb35590debf9bab27d361e26d0af38a774009c794eef"} Oct 30 16:41:05.892336 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.892307 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" event={"ID":"59f491b6-5fc5-4372-9a50-d60c3d5d6774","Type":"ContainerStarted","Data":"04f0cdd2cdd154abdf3fcf11e646ccb3500528f90fd2796d0590f125cd581cf5"} Oct 30 16:41:05.892445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:05.892386 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" event={"ID":"59f491b6-5fc5-4372-9a50-d60c3d5d6774","Type":"ContainerStarted","Data":"c1fac2e6ae0d0a8d2df9081f74b6d606e4f787b66bfad21d19d7ff5327964dea"} Oct 30 16:41:06.006287 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:06.006231 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" podStartSLOduration=2.006215143 podStartE2EDuration="2.006215143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:05.962831499 +0000 UTC m=+1175.113112487" watchObservedRunningTime="2026-10-30 16:41:06.006215143 +0000 UTC m=+1175.156496138" Oct 30 16:41:06.007694 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:06.007654 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" podStartSLOduration=2.007640749 podStartE2EDuration="2.007640749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:06.004897935 +0000 UTC m=+1175.155178924" watchObservedRunningTime="2026-10-30 16:41:06.007640749 +0000 UTC m=+1175.157921736" Oct 30 16:41:06.735640 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:06.734207 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:06.763780 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:06.757734 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:41:06.763780 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:06.757802 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" podUID="a965f27d-01a1-4d25-9331-91fe73051186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 16:41:06.809412 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:06.809378 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:06.821628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:06.810825 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:41:06.821628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:06.810887 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:41:07.740630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:07.727818 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:41:07.740630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:07.727890 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" podUID="a965f27d-01a1-4d25-9331-91fe73051186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 16:41:07.807120 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:07.807087 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:41:07.807286 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:07.807137 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:41:08.742628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:08.732643 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:41:08.742628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:08.732704 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" podUID="a965f27d-01a1-4d25-9331-91fe73051186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 16:41:08.807403 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:08.807364 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:41:08.807553 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:08.807419 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:41:09.729193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:09.728669 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:41:09.729193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:09.728731 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" podUID="a965f27d-01a1-4d25-9331-91fe73051186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 16:41:09.806874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:09.806835 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:41:09.807267 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:09.806892 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:41:10.731691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:10.731425 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:41:10.731691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:10.731502 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" podUID="a965f27d-01a1-4d25-9331-91fe73051186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 16:41:10.807307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:10.807273 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:41:10.807792 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:10.807767 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:41:11.725165 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:11.725131 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 30 16:41:11.725458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:11.725433 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" podUID="a965f27d-01a1-4d25-9331-91fe73051186" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 30 16:41:11.806858 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:11.806826 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 30 16:41:11.807050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:11.806886 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 30 16:41:12.841946 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:12.841912 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:12.843112 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:12.843082 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:12.844576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:12.844557 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:41:13.337076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:13.337007 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:13.337695 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:13.337668 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:13.338439 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:13.338422 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:41:14.364078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:14.364047 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:14.378197 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:14.378168 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:15.234889 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:15.234861 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:19.241461 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.241423 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x"] Oct 30 16:41:19.246251 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.246225 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.267532 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.267500 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x"] Oct 30 16:41:19.272211 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.272177 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f654570d-serving-cert\"" Oct 30 16:41:19.297720 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.297686 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-metrics-certs\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.298044 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.298023 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a380dfea-b521-412f-a42d-46ce64ed0427-openshift-service-ca\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.298340 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.298316 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-maas-api-tls\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.298468 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.298388 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckhqk\" (UniqueName: \"kubernetes.io/projected/a380dfea-b521-412f-a42d-46ce64ed0427-kube-api-access-ckhqk\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.399816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.399781 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-metrics-certs\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.400006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.399870 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a380dfea-b521-412f-a42d-46ce64ed0427-openshift-service-ca\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.400066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.400009 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-maas-api-tls\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.400121 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.400068 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckhqk\" (UniqueName: \"kubernetes.io/projected/a380dfea-b521-412f-a42d-46ce64ed0427-kube-api-access-ckhqk\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.400637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.400591 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a380dfea-b521-412f-a42d-46ce64ed0427-openshift-service-ca\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.402551 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.402524 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-metrics-certs\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.402902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.402881 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-maas-api-tls\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.421425 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.421401 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckhqk\" (UniqueName: \"kubernetes.io/projected/a380dfea-b521-412f-a42d-46ce64ed0427-kube-api-access-ckhqk\") pod \"maas-api-e2e-ait-f654570d-547fb94d97-tsl5x\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.562864 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.562784 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:19.564483 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.564454 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd"] Oct 30 16:41:19.568831 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.568813 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.597963 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.597843 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd"] Oct 30 16:41:19.605259 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.603761 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.605259 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.603836 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.605259 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.603924 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt2pl\" (UniqueName: \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-kube-api-access-pt2pl\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.605259 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.603999 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.605259 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.604031 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.613968 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.606262 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.614521 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.614486 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.614822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.614796 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.615041 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.615025 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.717107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.716753 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.717107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.716825 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt2pl\" (UniqueName: \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-kube-api-access-pt2pl\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.717107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.716865 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.717107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.716886 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.717107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.716971 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.717107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.717040 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.717107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.717077 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.717107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.717096 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.717715 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.717151 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.717778 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.717735 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.718556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.717903 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-certs\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.718556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.718080 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istiod-ca-cert\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.718556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.718427 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-data\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.718556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.718489 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-credential-socket\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.720105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.720087 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-podinfo\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.720475 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.720426 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-envoy\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.738345 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.738321 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-token\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.738577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.738558 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt2pl\" (UniqueName: \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-kube-api-access-pt2pl\") pod \"e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:19.907373 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.907073 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x"] Oct 30 16:41:19.907373 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:41:19.907180 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda380dfea_b521_412f_a42d_46ce64ed0427.slice/crio-6a9b6d771231926343f9947f8127f6f28d683f8c92c0c52b8961131442f4d21a WatchSource:0}: Error finding container 6a9b6d771231926343f9947f8127f6f28d683f8c92c0c52b8961131442f4d21a: Status 404 returned error can't find the container with id 6a9b6d771231926343f9947f8127f6f28d683f8c92c0c52b8961131442f4d21a Oct 30 16:41:19.926844 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:19.926806 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:20.283303 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:41:20.283260 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89880f7f_514c_4cbf_b623_c0ab7c16d8a7.slice/crio-2fa49904de572e9d5ee1d6186d15dbfef9476785806fc7183830bbe60bafbfe5 WatchSource:0}: Error finding container 2fa49904de572e9d5ee1d6186d15dbfef9476785806fc7183830bbe60bafbfe5: Status 404 returned error can't find the container with id 2fa49904de572e9d5ee1d6186d15dbfef9476785806fc7183830bbe60bafbfe5 Oct 30 16:41:20.285516 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:20.285482 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" event={"ID":"a380dfea-b521-412f-a42d-46ce64ed0427","Type":"ContainerStarted","Data":"6a9b6d771231926343f9947f8127f6f28d683f8c92c0c52b8961131442f4d21a"} Oct 30 16:41:20.291392 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:20.291340 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:20.291507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:20.291427 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:20.291507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:20.291459 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:41:20.291753 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:20.291730 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd"] Oct 30 16:41:21.294073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:21.294034 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" event={"ID":"a380dfea-b521-412f-a42d-46ce64ed0427","Type":"ContainerStarted","Data":"2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc"} Oct 30 16:41:21.294533 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:21.294127 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:21.295889 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:21.295864 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" event={"ID":"89880f7f-514c-4cbf-b623-c0ab7c16d8a7","Type":"ContainerStarted","Data":"1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff"} Oct 30 16:41:21.295992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:21.295905 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" event={"ID":"89880f7f-514c-4cbf-b623-c0ab7c16d8a7","Type":"ContainerStarted","Data":"2fa49904de572e9d5ee1d6186d15dbfef9476785806fc7183830bbe60bafbfe5"} Oct 30 16:41:21.356152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:21.356069 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" podStartSLOduration=1.943709702 podStartE2EDuration="2.356048788s" totalImagesPullingTime="412.339086ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:21.336148398 +0000 UTC m=+1190.486429387" watchObservedRunningTime="2026-10-30 16:41:21.356048788 +0000 UTC m=+1190.506329771" Oct 30 16:41:21.385855 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:21.385804 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podStartSLOduration=2.385784205 podStartE2EDuration="2.385784205s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:41:21.383204933 +0000 UTC m=+1190.533485923" watchObservedRunningTime="2026-10-30 16:41:21.385784205 +0000 UTC m=+1190.536065197" Oct 30 16:41:21.930627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:21.929974 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:21.936494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:21.936461 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:41:21.936695 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:21.936540 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:41:22.933637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:22.933115 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:41:22.933637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:22.933215 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:41:23.935066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:23.935027 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:41:23.935713 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:23.935677 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:41:24.723662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:24.723433 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 16:41:24.724146 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:24.723911 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:41:24.735653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:24.732408 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:24.935111 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:24.935072 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:41:24.935747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:24.935693 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:41:24.943297 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:41:24.943268 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:41:25.002298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.002252 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 30 16:41:25.002659 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.002631 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 16:41:25.039940 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.039897 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.040640 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.040178 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.220011 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.219964 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.225078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.225040 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.381821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.381709 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.382458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.382422 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/health\": context deadline exceeded" Oct 30 16:41:25.383529 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.383501 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.391195 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.391164 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.421626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.420712 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.438629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.434621 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.479206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.467377 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.735275 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.735224 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.767368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.744922 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:41:25.767368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.741854 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.767368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.741791 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.767368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.746374 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:41:25.767368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.746425 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:41:25.808897 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.808779 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.834629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.831733 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:25.920628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:25.917009 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.014685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.007942 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.014685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.008365 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:41:26.068759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.068708 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.069701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.069670 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": context deadline exceeded" Oct 30 16:41:26.077979 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.077956 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:41:26.103631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.078174 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:41:26.104432 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.082329 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:41:26.104668 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.104639 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:41:26.228659 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.225709 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.228659 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.227031 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.228659 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.228114 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:26.228659 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.228218 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:41:26.415147 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.415054 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.434290 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.434262 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:41:26.464708 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.464679 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 16:41:26.464964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.442723 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.639666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.443971 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.639666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.469776 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:41:26.639666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.469825 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:41:26.639666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.448044 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.639666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.461694 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.639666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.492942 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:41:26.639666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.492999 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:41:26.639666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.493034 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 16:41:26.639666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.569076 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.639666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.588405 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:41:26.660710 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.660680 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 16:41:26.754409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.754375 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:41:26.756042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.756000 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:41:26.761905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.758856 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:41:26.761905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.761689 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:26.957273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.955058 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:41:26.957273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:26.955129 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:41:27.031682 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.028641 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.061452 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.054841 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:27.061452 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.054909 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.080672 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.075348 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 16:41:27.226868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.226823 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 16:41:27.407510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.407412 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:41:27.419826 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.419786 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.498520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.498481 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:41:27.529500 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.529454 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.581626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.578845 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 16:41:27.609063 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.583720 6400 patch_prober.go:29] interesting pod/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:41:27.609063 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.604470 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.617253 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.617211 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.691765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.691653 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 16:41:27.755693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.755649 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.756721 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.756690 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.765238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.765203 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.765724 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.765702 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:27.766074 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.766055 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded" Oct 30 16:41:27.977130 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.768004 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 16:41:27.977442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.977420 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:41:28.004177 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.992218 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:41:28.004177 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.992277 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:41:28.021883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:27.963432 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:28.087635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.049108 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:28.306673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.057867 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:28.306673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.082959 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": context deadline exceeded" Oct 30 16:41:28.306673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.284775 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:28.306673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.302270 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r"] Oct 30 16:41:28.306673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.306500 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a45e6fe7-dxm7r"] Oct 30 16:41:28.536467 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.536422 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 16:41:28.565446 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.565358 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:41:28.569954 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.569929 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:41:28.572251 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.572213 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 16:41:28.577759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.572468 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 16:41:28.586146 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.586124 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 16:41:28.586765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.586726 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:41:28.586891 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.586792 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:41:28.587444 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.587424 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 16:41:28.944669 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.943802 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:41:28.944669 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.943857 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:41:28.970059 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:28.970026 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9da0e5b6-71e3-47cc-adee-e98bd2013137" path="/var/lib/kubelet/pods/9da0e5b6-71e3-47cc-adee-e98bd2013137/volumes" Oct 30 16:41:29.927791 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:29.927754 6400 patch_prober.go:29] interesting pod/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 30 16:41:29.928267 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:29.927824 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 30 16:41:30.932305 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:30.932276 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:30.932758 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:30.932741 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:30.933566 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:30.933537 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:41:51.427763 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:51.427704 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:51.574628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:51.573709 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:51.654633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:51.648301 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 30 16:41:51.684345 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:51.684182 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 16:41:51.789632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:51.774720 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:51.789632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:51.775359 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:41:51.789632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:51.775659 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:41:52.066789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.066693 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.108531 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.108486 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.391330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.390831 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.453779 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.453735 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.727458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.727360 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": context deadline exceeded" Oct 30 16:41:52.791031 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.778579 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.796704 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.778634 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.871076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.778656 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 30 16:41:52.871537 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.803135 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:41:52.886341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.886316 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:41:52.891895 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.891842 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:52.892364 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:52.892340 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:41:53.100424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:53.100384 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 16:41:53.110120 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:53.110070 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:41:53.137069 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:53.137029 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:41:53.137338 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:53.137325 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:41:53.567717 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:53.567676 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 16:41:53.568217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:53.568066 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:41:53.570996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:53.570975 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 16:41:53.582339 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:53.573772 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:41:53.582339 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:53.574725 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 16:41:54.602978 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:41:54.602946 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:41:59.509498 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.509466 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw"] Oct 30 16:41:59.514380 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.514360 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.543244 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.543209 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw"] Oct 30 16:41:59.553418 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.553393 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a475e0dd-serving-cert\"" Oct 30 16:41:59.576079 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.576046 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.576919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.576893 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7bm2\" (UniqueName: \"kubernetes.io/projected/464d29f0-a0f7-4644-925c-bee309ecf68e-kube-api-access-k7bm2\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.577193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.577129 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464d29f0-a0f7-4644-925c-bee309ecf68e-openshift-service-ca\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.577450 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.577429 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-metrics-certs\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.679709 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.679678 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.679908 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.679725 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7bm2\" (UniqueName: \"kubernetes.io/projected/464d29f0-a0f7-4644-925c-bee309ecf68e-kube-api-access-k7bm2\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.679908 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.679749 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464d29f0-a0f7-4644-925c-bee309ecf68e-openshift-service-ca\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.679908 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:41:59.679842 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-serving-cert: secret "maas-api-e2e-migrate-a475e0dd-serving-cert" not found Oct 30 16:41:59.679908 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.679849 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-metrics-certs\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.680129 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:41:59.679912 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls podName:464d29f0-a0f7-4644-925c-bee309ecf68e nodeName:}" failed. No retries permitted until 2026-10-30 16:42:00.179892084 +0000 UTC m=+1229.330173076 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls") pod "maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" (UID: "464d29f0-a0f7-4644-925c-bee309ecf68e") : secret "maas-api-e2e-migrate-a475e0dd-serving-cert" not found Oct 30 16:41:59.680637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.680582 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464d29f0-a0f7-4644-925c-bee309ecf68e-openshift-service-ca\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.682522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.682496 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-metrics-certs\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:41:59.701688 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:41:59.701660 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7bm2\" (UniqueName: \"kubernetes.io/projected/464d29f0-a0f7-4644-925c-bee309ecf68e-kube-api-access-k7bm2\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:42:00.189364 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:00.189324 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:42:00.189860 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:42:00.189836 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-serving-cert: secret "maas-api-e2e-migrate-a475e0dd-serving-cert" not found Oct 30 16:42:00.190036 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:42:00.189955 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls podName:464d29f0-a0f7-4644-925c-bee309ecf68e nodeName:}" failed. No retries permitted until 2026-10-30 16:42:01.18993357 +0000 UTC m=+1230.340214556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls") pod "maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" (UID: "464d29f0-a0f7-4644-925c-bee309ecf68e") : secret "maas-api-e2e-migrate-a475e0dd-serving-cert" not found Oct 30 16:42:01.205227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:01.205188 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:42:01.209146 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:01.208968 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls\") pod \"maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:42:01.328759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:01.328724 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:42:01.634741 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:01.634705 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw"] Oct 30 16:42:01.635414 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:42:01.635389 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464d29f0_a0f7_4644_925c_bee309ecf68e.slice/crio-a54f2e7511982a1ca84995a6ca0e0342f72e47df3eb79e6d4eeacb1e79f64b50 WatchSource:0}: Error finding container a54f2e7511982a1ca84995a6ca0e0342f72e47df3eb79e6d4eeacb1e79f64b50: Status 404 returned error can't find the container with id a54f2e7511982a1ca84995a6ca0e0342f72e47df3eb79e6d4eeacb1e79f64b50 Oct 30 16:42:01.637362 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:01.637345 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:42:02.516809 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.516768 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" event={"ID":"464d29f0-a0f7-4644-925c-bee309ecf68e","Type":"ContainerStarted","Data":"ee658c425d3d46fb464d7c4094964614cec0cf264b967b9d5108a691ad8b9889"} Oct 30 16:42:02.517230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.516827 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" event={"ID":"464d29f0-a0f7-4644-925c-bee309ecf68e","Type":"ContainerStarted","Data":"a54f2e7511982a1ca84995a6ca0e0342f72e47df3eb79e6d4eeacb1e79f64b50"} Oct 30 16:42:02.517230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.516959 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:42:02.556237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.556181 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" podStartSLOduration=3.17597375 podStartE2EDuration="3.556166199s" totalImagesPullingTime="380.192449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:41:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:02.553492836 +0000 UTC m=+1231.703773824" watchObservedRunningTime="2026-10-30 16:42:02.556166199 +0000 UTC m=+1231.706447188" Oct 30 16:42:02.926447 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.926364 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp"] Oct 30 16:42:02.926780 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.926583 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" containerID="cri-o://d844ec743c3bfa6d72c0972622f807b8ad8ecdb2ffa6d1f57e6da1c628c6c016" gracePeriod=30 Oct 30 16:42:02.930347 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.930320 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg"] Oct 30 16:42:02.930536 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.930514 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" containerID="cri-o://d4a479681567857b098df8ac71c8b0d1a3cfb29e0aaf01519184b91a8fa8a8c8" gracePeriod=30 Oct 30 16:42:02.935973 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.935941 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw"] Oct 30 16:42:02.936192 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.936167 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" containerID="cri-o://3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5" gracePeriod=30 Oct 30 16:42:02.939442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.939421 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj"] Oct 30 16:42:02.939625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:02.939586 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerName="istio-proxy" containerID="cri-o://83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa" gracePeriod=30 Oct 30 16:42:03.035015 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:03.034986 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5fbpp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:42:03.035172 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:03.035033 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:42:04.391635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.391575 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h"] Oct 30 16:42:04.396053 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.396029 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.427757 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.427719 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h"] Oct 30 16:42:04.453860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.453820 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.454053 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.453883 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.454124 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.454101 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.454238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.454202 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.454368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.454272 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.454368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.454330 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.454496 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.454386 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.454496 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.454438 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.454496 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.454469 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5lr2\" (UniqueName: \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-kube-api-access-n5lr2\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.555701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.555665 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.555924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.555718 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5lr2\" (UniqueName: \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-kube-api-access-n5lr2\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.555924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.555765 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.555924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.555790 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.555924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.555910 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.556157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.555961 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.556157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.556012 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.556157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.556140 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.556313 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.556179 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.556313 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.556254 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.556434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.556304 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-certs\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.556434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.556394 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-credential-socket\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.556538 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.556461 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-data\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.556832 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.556813 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istiod-ca-cert\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.558765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.558745 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-envoy\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.559116 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.559096 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-podinfo\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.577995 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.577968 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5lr2\" (UniqueName: \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-kube-api-access-n5lr2\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.580839 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.580806 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-token\") pod \"e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:04.710944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:04.710853 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:05.053214 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:05.053186 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h"] Oct 30 16:42:05.053712 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:42:05.053687 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5bc8d59_5d70_4dc7_954b_6c7310187f19.slice/crio-6d6daae2cd2862d00a532f9e146061dee5d59e88d1c31093d94ed2db8ec96e93 WatchSource:0}: Error finding container 6d6daae2cd2862d00a532f9e146061dee5d59e88d1c31093d94ed2db8ec96e93: Status 404 returned error can't find the container with id 6d6daae2cd2862d00a532f9e146061dee5d59e88d1c31093d94ed2db8ec96e93 Oct 30 16:42:05.056591 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:05.056558 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:05.056728 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:05.056691 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:05.056728 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:05.056724 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:05.546673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:05.546640 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" event={"ID":"e5bc8d59-5d70-4dc7-954b-6c7310187f19","Type":"ContainerStarted","Data":"832bf20a6d6d9a20e3f151d04e14d300adf3fc34cccb6a9be05aeec255fe06f1"} Oct 30 16:42:05.547074 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:05.546690 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" event={"ID":"e5bc8d59-5d70-4dc7-954b-6c7310187f19","Type":"ContainerStarted","Data":"6d6daae2cd2862d00a532f9e146061dee5d59e88d1c31093d94ed2db8ec96e93"} Oct 30 16:42:05.590777 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:05.590727 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" podStartSLOduration=1.590708421 podStartE2EDuration="1.590708421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:05.588569226 +0000 UTC m=+1234.738850215" watchObservedRunningTime="2026-10-30 16:42:05.590708421 +0000 UTC m=+1234.740989410" Oct 30 16:42:05.711034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:05.711001 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:06.731632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:06.730738 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 16:42:06.731632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:06.730810 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 16:42:07.734638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:07.731021 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 16:42:07.734638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:07.731080 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 16:42:08.489203 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.489031 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:42:08.508546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.508459 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:42:08.594320 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.586806 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:42:08.662804 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.662759 6400 generic.go:356] "Generic (PLEG): container finished" podID="208934b7-c63d-4074-a8bc-85d627b145df" containerID="d844ec743c3bfa6d72c0972622f807b8ad8ecdb2ffa6d1f57e6da1c628c6c016" exitCode=0 Oct 30 16:42:08.663118 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.663009 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" event={"ID":"208934b7-c63d-4074-a8bc-85d627b145df","Type":"ContainerDied","Data":"d844ec743c3bfa6d72c0972622f807b8ad8ecdb2ffa6d1f57e6da1c628c6c016"} Oct 30 16:42:08.668122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.668099 6400 generic.go:356] "Generic (PLEG): container finished" podID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerID="d4a479681567857b098df8ac71c8b0d1a3cfb29e0aaf01519184b91a8fa8a8c8" exitCode=0 Oct 30 16:42:08.668433 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.668416 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" event={"ID":"763ed816-0c8a-4043-ba58-636b1b9f9570","Type":"ContainerDied","Data":"d4a479681567857b098df8ac71c8b0d1a3cfb29e0aaf01519184b91a8fa8a8c8"} Oct 30 16:42:08.676204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.676176 6400 generic.go:356] "Generic (PLEG): container finished" podID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerID="3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5" exitCode=0 Oct 30 16:42:08.676442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.676424 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" event={"ID":"0349924c-b2b5-4a40-b6aa-9206c8046bc8","Type":"ContainerDied","Data":"3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5"} Oct 30 16:42:08.676574 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.676561 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" event={"ID":"0349924c-b2b5-4a40-b6aa-9206c8046bc8","Type":"ContainerDied","Data":"c7ff0f80ccc5718eb31b3b28f91058dfd9636edcdb8177d31bcf982b14836e2f"} Oct 30 16:42:08.676722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.676709 6400 scope.go:122] "RemoveContainer" containerID="3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5" Oct 30 16:42:08.676991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.676973 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw" Oct 30 16:42:08.679109 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679085 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-kube-api-access-t9x94\" (UniqueName: \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-kube-api-access-t9x94\") pod \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " Oct 30 16:42:08.679219 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679148 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-credential-socket\") pod \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " Oct 30 16:42:08.679219 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679197 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-certs\") pod \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " Oct 30 16:42:08.679341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679218 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-envoy\") pod \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " Oct 30 16:42:08.679341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679264 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istiod-ca-cert\") pod \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " Oct 30 16:42:08.679341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679289 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-token\" (UniqueName: \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-token\") pod \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " Oct 30 16:42:08.679497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679347 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-certs\") pod \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " Oct 30 16:42:08.679497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679366 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-data\") pod \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " Oct 30 16:42:08.679497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679420 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-token\" (UniqueName: \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-token\") pod \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " Oct 30 16:42:08.679497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679459 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-kube-api-access-kwsp5\" (UniqueName: \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-kube-api-access-kwsp5\") pod \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " Oct 30 16:42:08.679497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679485 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-socket\") pod \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " Oct 30 16:42:08.679784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679526 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-credential-socket\") pod \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " Oct 30 16:42:08.679784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679558 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-socket\") pod \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " Oct 30 16:42:08.679784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679591 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istiod-ca-cert\") pod \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " Oct 30 16:42:08.679784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679660 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-podinfo\") pod \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " Oct 30 16:42:08.679784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679680 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-podinfo\") pod \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " Oct 30 16:42:08.679784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679697 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-data\") pod \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\" (UID: \"0349924c-b2b5-4a40-b6aa-9206c8046bc8\") " Oct 30 16:42:08.679784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.679757 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-envoy\") pod \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\" (UID: \"c4d27ba0-2467-4de7-bf62-a064e2279d1f\") " Oct 30 16:42:08.681542 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.681515 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-socket" pod "c4d27ba0-2467-4de7-bf62-a064e2279d1f" (UID: "c4d27ba0-2467-4de7-bf62-a064e2279d1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.683435 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.682348 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-data" pod "0349924c-b2b5-4a40-b6aa-9206c8046bc8" (UID: "0349924c-b2b5-4a40-b6aa-9206c8046bc8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.683435 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.683185 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-data" pod "c4d27ba0-2467-4de7-bf62-a064e2279d1f" (UID: "c4d27ba0-2467-4de7-bf62-a064e2279d1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.683619 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.683502 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-credential-socket" pod "c4d27ba0-2467-4de7-bf62-a064e2279d1f" (UID: "c4d27ba0-2467-4de7-bf62-a064e2279d1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.684043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.683862 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-certs" pod "c4d27ba0-2467-4de7-bf62-a064e2279d1f" (UID: "c4d27ba0-2467-4de7-bf62-a064e2279d1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.684386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.684359 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-kube-api-access-t9x94" pod "0349924c-b2b5-4a40-b6aa-9206c8046bc8" (UID: "0349924c-b2b5-4a40-b6aa-9206c8046bc8"). InnerVolumeSpecName "kube-api-access-t9x94". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:08.685115 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.684928 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-certs" pod "0349924c-b2b5-4a40-b6aa-9206c8046bc8" (UID: "0349924c-b2b5-4a40-b6aa-9206c8046bc8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.685529 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.685504 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-socket" pod "0349924c-b2b5-4a40-b6aa-9206c8046bc8" (UID: "0349924c-b2b5-4a40-b6aa-9206c8046bc8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.685965 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.685943 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istiod-ca-cert" pod "c4d27ba0-2467-4de7-bf62-a064e2279d1f" (UID: "c4d27ba0-2467-4de7-bf62-a064e2279d1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:08.686641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.686618 6400 generic.go:356] "Generic (PLEG): container finished" podID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerID="83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa" exitCode=0 Oct 30 16:42:08.686902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.686882 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" event={"ID":"c4d27ba0-2467-4de7-bf62-a064e2279d1f","Type":"ContainerDied","Data":"83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa"} Oct 30 16:42:08.687072 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.687047 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" event={"ID":"c4d27ba0-2467-4de7-bf62-a064e2279d1f","Type":"ContainerDied","Data":"305e84ea6971a0ed3d398d992730ddfbf059898b4c3abb85084737b5d8604cf3"} Oct 30 16:42:08.687187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.687110 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-podinfo" pod "c4d27ba0-2467-4de7-bf62-a064e2279d1f" (UID: "c4d27ba0-2467-4de7-bf62-a064e2279d1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:42:08.687187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.687140 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj" Oct 30 16:42:08.687187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.687175 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-credential-socket" pod "0349924c-b2b5-4a40-b6aa-9206c8046bc8" (UID: "0349924c-b2b5-4a40-b6aa-9206c8046bc8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.687454 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.687434 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istiod-ca-cert" pod "0349924c-b2b5-4a40-b6aa-9206c8046bc8" (UID: "0349924c-b2b5-4a40-b6aa-9206c8046bc8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:08.688661 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.688619 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-token" pod "0349924c-b2b5-4a40-b6aa-9206c8046bc8" (UID: "0349924c-b2b5-4a40-b6aa-9206c8046bc8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:08.688984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.688941 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-envoy" pod "0349924c-b2b5-4a40-b6aa-9206c8046bc8" (UID: "0349924c-b2b5-4a40-b6aa-9206c8046bc8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.689068 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.689034 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-envoy" pod "c4d27ba0-2467-4de7-bf62-a064e2279d1f" (UID: "c4d27ba0-2467-4de7-bf62-a064e2279d1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.689995 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.689789 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-kube-api-access-kwsp5" pod "c4d27ba0-2467-4de7-bf62-a064e2279d1f" (UID: "c4d27ba0-2467-4de7-bf62-a064e2279d1f"). InnerVolumeSpecName "kube-api-access-kwsp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:08.690389 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.690366 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-podinfo" pod "0349924c-b2b5-4a40-b6aa-9206c8046bc8" (UID: "0349924c-b2b5-4a40-b6aa-9206c8046bc8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:42:08.690886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.690863 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-token" pod "c4d27ba0-2467-4de7-bf62-a064e2279d1f" (UID: "c4d27ba0-2467-4de7-bf62-a064e2279d1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:08.697558 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.696905 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:42:08.699793 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.699727 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:42:08.704078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.704059 6400 scope.go:122] "RemoveContainer" containerID="46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277" Oct 30 16:42:08.743141 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.743111 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 16:42:08.743727 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.743693 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 16:42:08.747821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.747798 6400 scope.go:122] "RemoveContainer" containerID="3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5" Oct 30 16:42:08.748306 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:42:08.748165 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5\": container with ID starting with 3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5 not found: ID does not exist" containerID="3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5" Oct 30 16:42:08.748306 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.748208 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5"} err="failed to get container status \"3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5\": rpc error: code = NotFound desc = could not find container \"3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5\": container with ID starting with 3735f31350239b52ff13d58cec6c446dac0bdb5299969f0b4e49cefd636384b5 not found: ID does not exist" Oct 30 16:42:08.748306 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.748227 6400 scope.go:122] "RemoveContainer" containerID="46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277" Oct 30 16:42:08.748635 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:42:08.748592 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277\": container with ID starting with 46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277 not found: ID does not exist" containerID="46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277" Oct 30 16:42:08.748716 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.748647 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277"} err="failed to get container status \"46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277\": rpc error: code = NotFound desc = could not find container \"46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277\": container with ID starting with 46b9bf5f1b8f4bae05ef603d33d5aaeacda6f36bb5edb8d5c51a8dc7f7ba3277 not found: ID does not exist" Oct 30 16:42:08.748716 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.748670 6400 scope.go:122] "RemoveContainer" containerID="83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa" Oct 30 16:42:08.762998 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.762978 6400 scope.go:122] "RemoveContainer" containerID="83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa" Oct 30 16:42:08.763453 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:42:08.763430 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa\": container with ID starting with 83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa not found: ID does not exist" containerID="83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa" Oct 30 16:42:08.764060 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.763584 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa"} err="failed to get container status \"83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa\": rpc error: code = NotFound desc = could not find container \"83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa\": container with ID starting with 83d5197495ac7b546ca7fb1ac39cf30d4fe89e9651aa27797e2df31b229f66aa not found: ID does not exist" Oct 30 16:42:08.784230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784197 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784233 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784246 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784259 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784272 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784284 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwsp5\" (UniqueName: \"kubernetes.io/projected/c4d27ba0-2467-4de7-bf62-a064e2279d1f-kube-api-access-kwsp5\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784297 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784311 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784324 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784336 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784347 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784360 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784372 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784384 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784396 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9x94\" (UniqueName: \"kubernetes.io/projected/0349924c-b2b5-4a40-b6aa-9206c8046bc8-kube-api-access-t9x94\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784408 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4d27ba0-2467-4de7-bf62-a064e2279d1f-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784423 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.784550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.784435 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0349924c-b2b5-4a40-b6aa-9206c8046bc8-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.886861 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.886701 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-kube-api-access-klsx9\" (UniqueName: \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-kube-api-access-klsx9\") pod \"763ed816-0c8a-4043-ba58-636b1b9f9570\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " Oct 30 16:42:08.887187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.887115 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-data\") pod \"763ed816-0c8a-4043-ba58-636b1b9f9570\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " Oct 30 16:42:08.887316 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.887287 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-socket\") pod \"208934b7-c63d-4074-a8bc-85d627b145df\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " Oct 30 16:42:08.887640 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.887558 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-socket" pod "208934b7-c63d-4074-a8bc-85d627b145df" (UID: "208934b7-c63d-4074-a8bc-85d627b145df"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.887756 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.887735 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-credential-socket\") pod \"763ed816-0c8a-4043-ba58-636b1b9f9570\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " Oct 30 16:42:08.888220 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.887888 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-data" pod "763ed816-0c8a-4043-ba58-636b1b9f9570" (UID: "763ed816-0c8a-4043-ba58-636b1b9f9570"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.888220 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888000 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-kube-api-access-gmcff\" (UniqueName: \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-kube-api-access-gmcff\") pod \"208934b7-c63d-4074-a8bc-85d627b145df\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " Oct 30 16:42:08.888220 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888107 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-data\") pod \"208934b7-c63d-4074-a8bc-85d627b145df\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " Oct 30 16:42:08.888220 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888154 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-certs\") pod \"763ed816-0c8a-4043-ba58-636b1b9f9570\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " Oct 30 16:42:08.888220 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888193 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-envoy\") pod \"208934b7-c63d-4074-a8bc-85d627b145df\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " Oct 30 16:42:08.888510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888226 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-token\" (UniqueName: \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-token\") pod \"763ed816-0c8a-4043-ba58-636b1b9f9570\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " Oct 30 16:42:08.888510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888259 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-certs\") pod \"208934b7-c63d-4074-a8bc-85d627b145df\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " Oct 30 16:42:08.888510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888295 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-podinfo\") pod \"763ed816-0c8a-4043-ba58-636b1b9f9570\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " Oct 30 16:42:08.888510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888333 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-istio-token\" (UniqueName: \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-istio-token\") pod \"208934b7-c63d-4074-a8bc-85d627b145df\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " Oct 30 16:42:08.888510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888415 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-envoy\") pod \"763ed816-0c8a-4043-ba58-636b1b9f9570\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " Oct 30 16:42:08.888510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888481 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/763ed816-0c8a-4043-ba58-636b1b9f9570-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763ed816-0c8a-4043-ba58-636b1b9f9570-istiod-ca-cert\") pod \"763ed816-0c8a-4043-ba58-636b1b9f9570\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " Oct 30 16:42:08.888924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888530 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-credential-socket\") pod \"208934b7-c63d-4074-a8bc-85d627b145df\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " Oct 30 16:42:08.888924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888644 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/208934b7-c63d-4074-a8bc-85d627b145df-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208934b7-c63d-4074-a8bc-85d627b145df-istiod-ca-cert\") pod \"208934b7-c63d-4074-a8bc-85d627b145df\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " Oct 30 16:42:08.888924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.888747 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-socket\") pod \"763ed816-0c8a-4043-ba58-636b1b9f9570\" (UID: \"763ed816-0c8a-4043-ba58-636b1b9f9570\") " Oct 30 16:42:08.888924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.887967 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-credential-socket" pod "763ed816-0c8a-4043-ba58-636b1b9f9570" (UID: "763ed816-0c8a-4043-ba58-636b1b9f9570"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.889138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.889011 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-certs" pod "208934b7-c63d-4074-a8bc-85d627b145df" (UID: "208934b7-c63d-4074-a8bc-85d627b145df"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.889416 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.889222 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-certs" pod "763ed816-0c8a-4043-ba58-636b1b9f9570" (UID: "763ed816-0c8a-4043-ba58-636b1b9f9570"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.889416 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.889366 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/208934b7-c63d-4074-a8bc-85d627b145df-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208934b7-c63d-4074-a8bc-85d627b145df-istio-podinfo\") pod \"208934b7-c63d-4074-a8bc-85d627b145df\" (UID: \"208934b7-c63d-4074-a8bc-85d627b145df\") " Oct 30 16:42:08.889549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.889433 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-credential-socket" pod "208934b7-c63d-4074-a8bc-85d627b145df" (UID: "208934b7-c63d-4074-a8bc-85d627b145df"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.889657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.889636 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-data" pod "208934b7-c63d-4074-a8bc-85d627b145df" (UID: "208934b7-c63d-4074-a8bc-85d627b145df"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.890233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.889945 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-kube-api-access-klsx9" pod "763ed816-0c8a-4043-ba58-636b1b9f9570" (UID: "763ed816-0c8a-4043-ba58-636b1b9f9570"). InnerVolumeSpecName "kube-api-access-klsx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:08.890233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890052 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/208934b7-c63d-4074-a8bc-85d627b145df-istiod-ca-cert" pod "208934b7-c63d-4074-a8bc-85d627b145df" (UID: "208934b7-c63d-4074-a8bc-85d627b145df"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:08.890233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890083 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/763ed816-0c8a-4043-ba58-636b1b9f9570-istiod-ca-cert" pod "763ed816-0c8a-4043-ba58-636b1b9f9570" (UID: "763ed816-0c8a-4043-ba58-636b1b9f9570"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890524 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/208934b7-c63d-4074-a8bc-85d627b145df-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890550 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-klsx9\" (UniqueName: \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-kube-api-access-klsx9\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890549 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-socket" pod "763ed816-0c8a-4043-ba58-636b1b9f9570" (UID: "763ed816-0c8a-4043-ba58-636b1b9f9570"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890564 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890578 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890596 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890632 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890646 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890662 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890674 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/763ed816-0c8a-4043-ba58-636b1b9f9570-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.890726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.890689 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.892295 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.892271 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/208934b7-c63d-4074-a8bc-85d627b145df-istio-podinfo" pod "208934b7-c63d-4074-a8bc-85d627b145df" (UID: "208934b7-c63d-4074-a8bc-85d627b145df"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:42:08.892862 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.892840 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-envoy" pod "763ed816-0c8a-4043-ba58-636b1b9f9570" (UID: "763ed816-0c8a-4043-ba58-636b1b9f9570"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.893200 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.893158 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-envoy" pod "208934b7-c63d-4074-a8bc-85d627b145df" (UID: "208934b7-c63d-4074-a8bc-85d627b145df"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:08.893706 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.893679 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-token" pod "763ed816-0c8a-4043-ba58-636b1b9f9570" (UID: "763ed816-0c8a-4043-ba58-636b1b9f9570"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:08.894896 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.894871 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-istio-token" pod "208934b7-c63d-4074-a8bc-85d627b145df" (UID: "208934b7-c63d-4074-a8bc-85d627b145df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:08.895098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.895063 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-podinfo" pod "763ed816-0c8a-4043-ba58-636b1b9f9570" (UID: "763ed816-0c8a-4043-ba58-636b1b9f9570"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:42:08.895426 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.895401 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-kube-api-access-gmcff" pod "208934b7-c63d-4074-a8bc-85d627b145df" (UID: "208934b7-c63d-4074-a8bc-85d627b145df"). InnerVolumeSpecName "kube-api-access-gmcff". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:08.992082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.991948 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmcff\" (UniqueName: \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-kube-api-access-gmcff\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.992082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.991971 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/208934b7-c63d-4074-a8bc-85d627b145df-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.992082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.991981 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.992082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.991989 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.992082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.991997 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/208934b7-c63d-4074-a8bc-85d627b145df-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.992082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.992006 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.992082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.992014 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/763ed816-0c8a-4043-ba58-636b1b9f9570-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:08.992082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:08.992051 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/208934b7-c63d-4074-a8bc-85d627b145df-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:09.041781 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.041723 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw"] Oct 30 16:42:09.046360 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.046333 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-xpkrw"] Oct 30 16:42:09.077159 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.077128 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj"] Oct 30 16:42:09.084043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.084016 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-srnkj"] Oct 30 16:42:09.702344 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.702311 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" Oct 30 16:42:09.703050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.702410 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp" event={"ID":"208934b7-c63d-4074-a8bc-85d627b145df","Type":"ContainerDied","Data":"954902855be85b5ed2266160c1f6c27d49580411634feeb1c9245773573dc091"} Oct 30 16:42:09.703050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.702478 6400 scope.go:122] "RemoveContainer" containerID="d844ec743c3bfa6d72c0972622f807b8ad8ecdb2ffa6d1f57e6da1c628c6c016" Oct 30 16:42:09.705482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.705421 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" event={"ID":"763ed816-0c8a-4043-ba58-636b1b9f9570","Type":"ContainerDied","Data":"4fba2bf7049901caa89253453fe81b58b0af5ea5da3027ecb869d80415a06703"} Oct 30 16:42:09.705775 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.705743 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg" Oct 30 16:42:09.718464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.718436 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 16:42:09.718738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.718710 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 16:42:09.763911 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.763882 6400 scope.go:122] "RemoveContainer" containerID="d4a479681567857b098df8ac71c8b0d1a3cfb29e0aaf01519184b91a8fa8a8c8" Oct 30 16:42:09.766994 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.766966 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp"] Oct 30 16:42:09.775161 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.775086 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5fbpp"] Oct 30 16:42:09.800236 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.800208 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg"] Oct 30 16:42:09.804893 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:09.804869 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lzfbg"] Oct 30 16:42:10.722147 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:10.722110 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 16:42:10.722435 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:10.722408 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 16:42:10.976200 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:10.976098 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" path="/var/lib/kubelet/pods/0349924c-b2b5-4a40-b6aa-9206c8046bc8/volumes" Oct 30 16:42:10.976701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:10.976691 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="208934b7-c63d-4074-a8bc-85d627b145df" path="/var/lib/kubelet/pods/208934b7-c63d-4074-a8bc-85d627b145df/volumes" Oct 30 16:42:10.977174 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:10.977159 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" path="/var/lib/kubelet/pods/763ed816-0c8a-4043-ba58-636b1b9f9570/volumes" Oct 30 16:42:10.977934 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:10.977920 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" path="/var/lib/kubelet/pods/c4d27ba0-2467-4de7-bf62-a064e2279d1f/volumes" Oct 30 16:42:11.712384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:11.712337 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 30 16:42:11.712615 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:11.712401 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 30 16:42:13.015199 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:13.015168 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:13.015685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:13.015617 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:13.016520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:13.016504 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:42:14.585698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:14.585666 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v"] Oct 30 16:42:14.586200 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:14.585887 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" containerID="cri-o://d46e54fdba2c9869b091e93d6c24db73c4e54ca62579e24a7c4567aa83cdaac1" gracePeriod=30 Oct 30 16:42:18.723287 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.723248 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5"] Oct 30 16:42:18.724490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.724461 6400 cpu_manager.go:414] "containerMap: removing container" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" Oct 30 16:42:18.724490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.724491 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" Oct 30 16:42:18.724713 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.724625 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" Oct 30 16:42:18.724713 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.724633 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" Oct 30 16:42:18.724829 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.724765 6400 cpu_manager.go:414] "containerMap: removing container" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" Oct 30 16:42:18.724829 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.724799 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" Oct 30 16:42:18.724927 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.724857 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerName="istio-proxy" Oct 30 16:42:18.724927 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.724865 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerName="istio-proxy" Oct 30 16:42:18.725336 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.725314 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="208934b7-c63d-4074-a8bc-85d627b145df" containerName="istio-proxy" Oct 30 16:42:18.725436 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.725349 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" Oct 30 16:42:18.725436 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.725365 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4d27ba0-2467-4de7-bf62-a064e2279d1f" containerName="istio-proxy" Oct 30 16:42:18.725436 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.725384 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="763ed816-0c8a-4043-ba58-636b1b9f9570" containerName="istio-proxy" Oct 30 16:42:18.753795 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.753765 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:18.756692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.756664 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5"] Oct 30 16:42:18.769174 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.769151 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-e4aca49c-openshift-default-dockercfg-xchw6\"" Oct 30 16:42:18.899084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.899050 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07713c9b-864c-43db-9a1b-f7aef356f607-istiod-ca-cert\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:18.899297 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.899130 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-data\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:18.899385 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.899298 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-envoy\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:18.899385 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.899373 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-certs\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:18.899511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.899485 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07713c9b-864c-43db-9a1b-f7aef356f607-istio-podinfo\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:18.899568 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.899549 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpg98\" (UniqueName: \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-kube-api-access-dpg98\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:18.899677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.899660 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-credential-socket\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:18.899799 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.899783 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-socket\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:18.899863 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:18.899821 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-istio-token\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.001386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.001350 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-socket\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.001594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.001406 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-istio-token\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.001594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.001453 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07713c9b-864c-43db-9a1b-f7aef356f607-istiod-ca-cert\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.001594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.001545 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-data\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.001791 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.001697 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-envoy\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.001870 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.001840 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-socket\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.002055 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.001854 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-certs\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.002169 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.002100 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07713c9b-864c-43db-9a1b-f7aef356f607-istio-podinfo\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.002169 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.001906 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-data\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.002169 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.002131 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-certs\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.002169 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.002151 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpg98\" (UniqueName: \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-kube-api-access-dpg98\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.002371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.002209 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-credential-socket\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.002371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.002264 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07713c9b-864c-43db-9a1b-f7aef356f607-istiod-ca-cert\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.002581 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.002531 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-credential-socket\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.005482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.005457 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07713c9b-864c-43db-9a1b-f7aef356f607-istio-podinfo\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.005745 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.005723 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-envoy\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.023557 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.023533 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpg98\" (UniqueName: \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-kube-api-access-dpg98\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.024041 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.024006 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-istio-token\") pod \"e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.079148 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.079111 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:19.436328 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.436296 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5"] Oct 30 16:42:19.437648 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:42:19.437572 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07713c9b_864c_43db_9a1b_f7aef356f607.slice/crio-2a0f4ab596a21dcb2447cc84769387b82090ebe34d8a50a5f0c6d18084b32b44 WatchSource:0}: Error finding container 2a0f4ab596a21dcb2447cc84769387b82090ebe34d8a50a5f0c6d18084b32b44: Status 404 returned error can't find the container with id 2a0f4ab596a21dcb2447cc84769387b82090ebe34d8a50a5f0c6d18084b32b44 Oct 30 16:42:19.440727 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.440694 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:19.440856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.440760 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:19.440856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.440798 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:19.868868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.868834 6400 generic.go:356] "Generic (PLEG): container finished" podID="608709cd-a150-41dd-ad88-ea0f0040de82" containerID="d46e54fdba2c9869b091e93d6c24db73c4e54ca62579e24a7c4567aa83cdaac1" exitCode=0 Oct 30 16:42:19.869341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.868914 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" event={"ID":"608709cd-a150-41dd-ad88-ea0f0040de82","Type":"ContainerDied","Data":"d46e54fdba2c9869b091e93d6c24db73c4e54ca62579e24a7c4567aa83cdaac1"} Oct 30 16:42:19.869341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.868995 6400 scope.go:122] "RemoveContainer" containerID="82b95fd9ba5127519f332fd844fd32932442a789bec5053913cd873a1a2e32ac" Oct 30 16:42:19.871504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.871469 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" event={"ID":"07713c9b-864c-43db-9a1b-f7aef356f607","Type":"ContainerStarted","Data":"847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513"} Oct 30 16:42:19.871647 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.871516 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" event={"ID":"07713c9b-864c-43db-9a1b-f7aef356f607","Type":"ContainerStarted","Data":"2a0f4ab596a21dcb2447cc84769387b82090ebe34d8a50a5f0c6d18084b32b44"} Oct 30 16:42:19.894072 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.894034 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:42:19.916464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:19.916399 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" podStartSLOduration=1.916380678 podStartE2EDuration="1.916380678s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:19.911918839 +0000 UTC m=+1249.062199830" watchObservedRunningTime="2026-10-30 16:42:19.916380678 +0000 UTC m=+1249.066661669" Oct 30 16:42:20.032778 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.032746 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-istio-token\" (UniqueName: \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-istio-token\") pod \"608709cd-a150-41dd-ad88-ea0f0040de82\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " Oct 30 16:42:20.032933 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.032832 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-kube-api-access-skxfn\" (UniqueName: \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-kube-api-access-skxfn\") pod \"608709cd-a150-41dd-ad88-ea0f0040de82\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " Oct 30 16:42:20.032933 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.032910 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-envoy\") pod \"608709cd-a150-41dd-ad88-ea0f0040de82\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " Oct 30 16:42:20.033019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.032940 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-socket\") pod \"608709cd-a150-41dd-ad88-ea0f0040de82\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " Oct 30 16:42:20.033069 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.033037 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/608709cd-a150-41dd-ad88-ea0f0040de82-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608709cd-a150-41dd-ad88-ea0f0040de82-istio-podinfo\") pod \"608709cd-a150-41dd-ad88-ea0f0040de82\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " Oct 30 16:42:20.033125 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.033109 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-data\") pod \"608709cd-a150-41dd-ad88-ea0f0040de82\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " Oct 30 16:42:20.033186 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.033142 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-credential-socket\") pod \"608709cd-a150-41dd-ad88-ea0f0040de82\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " Oct 30 16:42:20.033244 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.033228 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/608709cd-a150-41dd-ad88-ea0f0040de82-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608709cd-a150-41dd-ad88-ea0f0040de82-istiod-ca-cert\") pod \"608709cd-a150-41dd-ad88-ea0f0040de82\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " Oct 30 16:42:20.033297 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.033270 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-certs\") pod \"608709cd-a150-41dd-ad88-ea0f0040de82\" (UID: \"608709cd-a150-41dd-ad88-ea0f0040de82\") " Oct 30 16:42:20.033348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.033288 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-socket" pod "608709cd-a150-41dd-ad88-ea0f0040de82" (UID: "608709cd-a150-41dd-ad88-ea0f0040de82"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:20.033406 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.033377 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-data" pod "608709cd-a150-41dd-ad88-ea0f0040de82" (UID: "608709cd-a150-41dd-ad88-ea0f0040de82"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:20.033466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.033447 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-credential-socket" pod "608709cd-a150-41dd-ad88-ea0f0040de82" (UID: "608709cd-a150-41dd-ad88-ea0f0040de82"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:20.033701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.033667 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-certs" pod "608709cd-a150-41dd-ad88-ea0f0040de82" (UID: "608709cd-a150-41dd-ad88-ea0f0040de82"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:20.033701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.033689 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/608709cd-a150-41dd-ad88-ea0f0040de82-istiod-ca-cert" pod "608709cd-a150-41dd-ad88-ea0f0040de82" (UID: "608709cd-a150-41dd-ad88-ea0f0040de82"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:20.034229 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.034205 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.034317 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.034234 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.034317 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.034249 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.034317 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.034268 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/608709cd-a150-41dd-ad88-ea0f0040de82-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.034317 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.034281 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.036015 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.035960 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-istio-token" pod "608709cd-a150-41dd-ad88-ea0f0040de82" (UID: "608709cd-a150-41dd-ad88-ea0f0040de82"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:20.036110 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.036047 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-envoy" pod "608709cd-a150-41dd-ad88-ea0f0040de82" (UID: "608709cd-a150-41dd-ad88-ea0f0040de82"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:42:20.036233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.036215 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/608709cd-a150-41dd-ad88-ea0f0040de82-istio-podinfo" pod "608709cd-a150-41dd-ad88-ea0f0040de82" (UID: "608709cd-a150-41dd-ad88-ea0f0040de82"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:42:20.036367 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.036343 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-kube-api-access-skxfn" pod "608709cd-a150-41dd-ad88-ea0f0040de82" (UID: "608709cd-a150-41dd-ad88-ea0f0040de82"). InnerVolumeSpecName "kube-api-access-skxfn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:20.079394 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.079363 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:20.086542 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.086514 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:20.146987 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.146901 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.146987 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.146932 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skxfn\" (UniqueName: \"kubernetes.io/projected/608709cd-a150-41dd-ad88-ea0f0040de82-kube-api-access-skxfn\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.146987 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.146947 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/608709cd-a150-41dd-ad88-ea0f0040de82-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.146987 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.146959 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/608709cd-a150-41dd-ad88-ea0f0040de82-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:20.879811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.879785 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" Oct 30 16:42:20.880263 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.879823 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v" event={"ID":"608709cd-a150-41dd-ad88-ea0f0040de82","Type":"ContainerDied","Data":"af8bd9ffa09fe36302428df54643a8fdd6fcac72b26648922f1a26bb3ff4cfda"} Oct 30 16:42:20.880263 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.879881 6400 scope.go:122] "RemoveContainer" containerID="d46e54fdba2c9869b091e93d6c24db73c4e54ca62579e24a7c4567aa83cdaac1" Oct 30 16:42:20.880423 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.880397 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:20.881710 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.881691 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:42:20.939190 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.939156 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v"] Oct 30 16:42:20.948424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.948402 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-a8c22d-openshift-default-7c77486ff9-xz69v"] Oct 30 16:42:20.967685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:20.967634 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" path="/var/lib/kubelet/pods/608709cd-a150-41dd-ad88-ea0f0040de82/volumes" Oct 30 16:42:24.602892 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:42:24.602853 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:42:25.286643 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.286590 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x"] Oct 30 16:42:25.286916 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.286886 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" podUID="a380dfea-b521-412f-a42d-46ce64ed0427" containerName="maas-api" containerID="cri-o://2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc" gracePeriod=30 Oct 30 16:42:25.744154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.742642 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:42:25.852822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.852738 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-metrics-certs\") pod \"a380dfea-b521-412f-a42d-46ce64ed0427\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " Oct 30 16:42:25.853154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.853131 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a380dfea-b521-412f-a42d-46ce64ed0427-kube-api-access-ckhqk\" (UniqueName: \"kubernetes.io/projected/a380dfea-b521-412f-a42d-46ce64ed0427-kube-api-access-ckhqk\") pod \"a380dfea-b521-412f-a42d-46ce64ed0427\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " Oct 30 16:42:25.853431 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.853415 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a380dfea-b521-412f-a42d-46ce64ed0427-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a380dfea-b521-412f-a42d-46ce64ed0427-openshift-service-ca\") pod \"a380dfea-b521-412f-a42d-46ce64ed0427\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " Oct 30 16:42:25.854483 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.854460 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-maas-api-tls\") pod \"a380dfea-b521-412f-a42d-46ce64ed0427\" (UID: \"a380dfea-b521-412f-a42d-46ce64ed0427\") " Oct 30 16:42:25.854854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.854782 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a380dfea-b521-412f-a42d-46ce64ed0427-openshift-service-ca" pod "a380dfea-b521-412f-a42d-46ce64ed0427" (UID: "a380dfea-b521-412f-a42d-46ce64ed0427"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:42:25.855840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.855814 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a380dfea-b521-412f-a42d-46ce64ed0427-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:25.858534 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.858509 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-maas-api-tls" pod "a380dfea-b521-412f-a42d-46ce64ed0427" (UID: "a380dfea-b521-412f-a42d-46ce64ed0427"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:25.865946 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.865923 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-metrics-certs" pod "a380dfea-b521-412f-a42d-46ce64ed0427" (UID: "a380dfea-b521-412f-a42d-46ce64ed0427"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:42:25.884628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.870799 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a380dfea-b521-412f-a42d-46ce64ed0427-kube-api-access-ckhqk" pod "a380dfea-b521-412f-a42d-46ce64ed0427" (UID: "a380dfea-b521-412f-a42d-46ce64ed0427"). InnerVolumeSpecName "kube-api-access-ckhqk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:42:25.922811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.922776 6400 generic.go:356] "Generic (PLEG): container finished" podID="a380dfea-b521-412f-a42d-46ce64ed0427" containerID="2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc" exitCode=0 Oct 30 16:42:25.923015 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.922997 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" event={"ID":"a380dfea-b521-412f-a42d-46ce64ed0427","Type":"ContainerDied","Data":"2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc"} Oct 30 16:42:25.923083 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.923063 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" event={"ID":"a380dfea-b521-412f-a42d-46ce64ed0427","Type":"ContainerDied","Data":"6a9b6d771231926343f9947f8127f6f28d683f8c92c0c52b8961131442f4d21a"} Oct 30 16:42:25.923137 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.923087 6400 scope.go:122] "RemoveContainer" containerID="2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc" Oct 30 16:42:25.923223 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.923211 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x" Oct 30 16:42:25.940311 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.940279 6400 scope.go:122] "RemoveContainer" containerID="2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc" Oct 30 16:42:25.940735 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:42:25.940700 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc\": container with ID starting with 2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc not found: ID does not exist" containerID="2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc" Oct 30 16:42:25.940930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.940898 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc"} err="failed to get container status \"2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc\": rpc error: code = NotFound desc = could not find container \"2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc\": container with ID starting with 2adb98b365d3289b37dc4ac9ad1db9bfc3ae37fa7f5fc29c3f766b6b0a50f7cc not found: ID does not exist" Oct 30 16:42:25.963193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.963115 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:25.963193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.963149 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a380dfea-b521-412f-a42d-46ce64ed0427-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:25.963193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.963163 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckhqk\" (UniqueName: \"kubernetes.io/projected/a380dfea-b521-412f-a42d-46ce64ed0427-kube-api-access-ckhqk\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:42:25.972746 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.972697 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x"] Oct 30 16:42:25.978264 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:25.978234 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f654570d-547fb94d97-tsl5x"] Oct 30 16:42:26.969118 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:26.969083 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a380dfea-b521-412f-a42d-46ce64ed0427" path="/var/lib/kubelet/pods/a380dfea-b521-412f-a42d-46ce64ed0427/volumes" Oct 30 16:42:27.642656 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.642618 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt"] Oct 30 16:42:27.643567 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643542 6400 cpu_manager.go:414] "containerMap: removing container" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" Oct 30 16:42:27.643697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643569 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" Oct 30 16:42:27.643697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643616 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" Oct 30 16:42:27.643697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643625 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" Oct 30 16:42:27.643697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643648 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a380dfea-b521-412f-a42d-46ce64ed0427" containerName="maas-api" Oct 30 16:42:27.643697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643655 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a380dfea-b521-412f-a42d-46ce64ed0427" containerName="maas-api" Oct 30 16:42:27.643697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643665 6400 cpu_manager.go:414] "containerMap: removing container" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" Oct 30 16:42:27.643697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643672 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" Oct 30 16:42:27.643909 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643875 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0349924c-b2b5-4a40-b6aa-9206c8046bc8" containerName="istio-proxy" Oct 30 16:42:27.643909 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643888 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" Oct 30 16:42:27.643909 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.643906 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a380dfea-b521-412f-a42d-46ce64ed0427" containerName="maas-api" Oct 30 16:42:27.649207 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.649187 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.664114 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.664081 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt"] Oct 30 16:42:27.688267 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.688222 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-credential-socket\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.688455 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.688314 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-certs\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.688455 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.688355 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-podinfo\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.688455 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.688439 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-token\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.688675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.688527 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7b2s\" (UniqueName: \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-kube-api-access-r7b2s\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.688675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.688567 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istiod-ca-cert\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.688675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.688641 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-data\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.688845 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.688678 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-envoy\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.688908 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.688861 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-socket\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.708463 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.708440 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-a0777a0c-openshift-default-dockercfg-p2dsx\"" Oct 30 16:42:27.790068 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790036 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-token\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790240 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790130 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7b2s\" (UniqueName: \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-kube-api-access-r7b2s\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790240 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790167 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istiod-ca-cert\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790240 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790220 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-data\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790255 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-envoy\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790419 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-socket\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790466 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-credential-socket\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790540 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-certs\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790704 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790572 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-podinfo\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790763 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790732 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-data\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790866 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790840 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-socket\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.790999 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.790978 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-credential-socket\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.791060 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.791037 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istiod-ca-cert\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.791124 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.791041 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-certs\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.793248 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.793217 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-envoy\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.793341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.793298 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-podinfo\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.814062 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.814031 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7b2s\" (UniqueName: \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-kube-api-access-r7b2s\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.829648 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.827156 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-token\") pod \"e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:27.978697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:27.976042 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:28.331303 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:28.331276 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt"] Oct 30 16:42:28.332082 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:42:28.332058 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc07ad2d1_4d6f_491b_95b6_a5b9f9a83483.slice/crio-e26e8d83abe469892261cf63dba3784d54d622f20c712073d0af6d6a540afa47 WatchSource:0}: Error finding container e26e8d83abe469892261cf63dba3784d54d622f20c712073d0af6d6a540afa47: Status 404 returned error can't find the container with id e26e8d83abe469892261cf63dba3784d54d622f20c712073d0af6d6a540afa47 Oct 30 16:42:28.335460 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:28.335420 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:28.335539 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:28.335510 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:28.335575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:28.335553 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:28.953563 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:28.953513 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" event={"ID":"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483","Type":"ContainerStarted","Data":"a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18"} Oct 30 16:42:28.953864 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:28.953643 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" event={"ID":"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483","Type":"ContainerStarted","Data":"e26e8d83abe469892261cf63dba3784d54d622f20c712073d0af6d6a540afa47"} Oct 30 16:42:28.976514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:28.976482 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:29.982961 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:29.982931 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:30.036027 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:30.035936 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" podStartSLOduration=3.035921167 podStartE2EDuration="3.035921167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:29.017282976 +0000 UTC m=+1258.167563964" watchObservedRunningTime="2026-10-30 16:42:30.035921167 +0000 UTC m=+1259.186202159" Oct 30 16:42:30.973894 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:30.973866 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:30.974912 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:30.974888 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:42:31.215066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:31.215029 6400 scope.go:122] "RemoveContainer" containerID="d1992c4d3855c091b708330a9505c50fe9fa96f0711bf8a1e9630cbbac6bfbfc" Oct 30 16:42:37.426630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:37.425708 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:37.764630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:37.763633 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:42:37.784628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:37.771674 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:42:48.066823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.066748 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4"] Oct 30 16:42:48.067459 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.067441 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="608709cd-a150-41dd-ad88-ea0f0040de82" containerName="istio-proxy" Oct 30 16:42:48.071175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.071157 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.083700 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.083672 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q"] Oct 30 16:42:48.087881 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.087862 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk"] Oct 30 16:42:48.087992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.087970 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.089082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.089066 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nztr5\"" Oct 30 16:42:48.091844 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.091827 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.103107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.101193 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4"] Oct 30 16:42:48.158376 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.158343 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q"] Oct 30 16:42:48.162588 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.162561 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk"] Oct 30 16:42:48.184282 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.184247 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.184550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.184316 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gvb6k\" (UniqueName: \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-kube-api-access-gvb6k\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.184550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.184350 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/628cf0ef-f9e0-4469-a4aa-5528721b5818-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.184550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.184390 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.184550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.184423 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.184822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.184799 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.184994 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.184976 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.185082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185017 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.185082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185042 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.185181 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185093 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.185181 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185130 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.185181 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185169 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.185330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185189 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.185330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185220 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.185330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185258 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.185330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185300 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5lrl2\" (UniqueName: \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-kube-api-access-5lrl2\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.185522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185331 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.185522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185367 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.185522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185397 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.185522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185449 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.185522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185483 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.185522 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185504 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hcljr\" (UniqueName: \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-kube-api-access-hcljr\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.185823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185531 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.185823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185595 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.185823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185700 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.185823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185740 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.185823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.185767 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.286949 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.286917 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.287153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.286960 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.287153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.286987 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5lrl2\" (UniqueName: \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-kube-api-access-5lrl2\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.287153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287008 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.287153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287031 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.287153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287051 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.287153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287082 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.287153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287117 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.287572 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287390 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hcljr\" (UniqueName: \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-kube-api-access-hcljr\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.287572 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287436 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.287572 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287487 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.287572 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287497 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.287572 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287506 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.287572 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287508 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.287907 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287644 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.287907 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287692 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.287907 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287732 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.287907 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287796 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.287907 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287843 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.287907 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287857 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.288224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.287979 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gvb6k\" (UniqueName: \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-kube-api-access-gvb6k\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.288224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288017 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/628cf0ef-f9e0-4469-a4aa-5528721b5818-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.288224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288033 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.288224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288064 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.288224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288075 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.288224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288108 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.288224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288152 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.288224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288187 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.288651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288229 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.288651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288261 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.288651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288322 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.288651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288368 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.288651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288415 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.288651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288438 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.289157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288706 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.289157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288753 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/628cf0ef-f9e0-4469-a4aa-5528721b5818-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.289157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288185 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.289157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.288986 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.289157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.289045 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.289440 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.289224 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.289440 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.289305 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.290009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.289986 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.290517 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.290488 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.290805 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.290778 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.290903 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.290816 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.291381 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.291361 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.291544 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.291518 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.291702 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.291684 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.309587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.309561 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.310509 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.310451 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.310671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.310638 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.310931 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.310914 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gvb6k\" (UniqueName: \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-kube-api-access-gvb6k\") pod \"maas-default-gateway-openshift-default-76f4876667-gjgfk\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.311126 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.311105 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hcljr\" (UniqueName: \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-kube-api-access-hcljr\") pod \"maas-default-gateway-openshift-default-76f4876667-jfk8q\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.311236 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.311221 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5lrl2\" (UniqueName: \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-kube-api-access-5lrl2\") pod \"maas-default-gateway-openshift-default-76f4876667-4cwj4\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.386672 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.386578 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:48.407617 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.405821 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:48.495276 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.493864 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:48.743043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.743015 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4"] Oct 30 16:42:48.743141 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:42:48.743043 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod605b7c33_6b21_4aa8_bfa4_c85298b8b717.slice/crio-f8e232c3d0f8de38bf46a146535086eb3d0a799e89a7701d20dfb77bd882b83c WatchSource:0}: Error finding container f8e232c3d0f8de38bf46a146535086eb3d0a799e89a7701d20dfb77bd882b83c: Status 404 returned error can't find the container with id f8e232c3d0f8de38bf46a146535086eb3d0a799e89a7701d20dfb77bd882b83c Oct 30 16:42:48.746041 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.746009 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:48.746116 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.746074 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:48.746116 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.746104 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:48.758547 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.758526 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q"] Oct 30 16:42:48.764063 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:42:48.764037 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628cf0ef_f9e0_4469_a4aa_5528721b5818.slice/crio-b9fa02f6a44c5804359a1fa84513e924f209d695d0de3dd8fba3a36ee04bcafc WatchSource:0}: Error finding container b9fa02f6a44c5804359a1fa84513e924f209d695d0de3dd8fba3a36ee04bcafc: Status 404 returned error can't find the container with id b9fa02f6a44c5804359a1fa84513e924f209d695d0de3dd8fba3a36ee04bcafc Oct 30 16:42:48.766172 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.766142 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:48.766227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.766208 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:48.766264 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.766240 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:48.855066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.855035 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk"] Oct 30 16:42:48.862268 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.862212 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:48.862368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.862311 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:48.862408 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:48.862362 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:42:49.191567 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.191484 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" event={"ID":"628cf0ef-f9e0-4469-a4aa-5528721b5818","Type":"ContainerStarted","Data":"38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e"} Oct 30 16:42:49.192093 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.192066 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" event={"ID":"628cf0ef-f9e0-4469-a4aa-5528721b5818","Type":"ContainerStarted","Data":"b9fa02f6a44c5804359a1fa84513e924f209d695d0de3dd8fba3a36ee04bcafc"} Oct 30 16:42:49.193434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.193412 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" event={"ID":"605b7c33-6b21-4aa8-bfa4-c85298b8b717","Type":"ContainerStarted","Data":"b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2"} Oct 30 16:42:49.193540 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.193459 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" event={"ID":"605b7c33-6b21-4aa8-bfa4-c85298b8b717","Type":"ContainerStarted","Data":"f8e232c3d0f8de38bf46a146535086eb3d0a799e89a7701d20dfb77bd882b83c"} Oct 30 16:42:49.195386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.195361 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" event={"ID":"7fc3c304-52a0-4f96-8f6b-021c866e3f0a","Type":"ContainerStarted","Data":"efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722"} Oct 30 16:42:49.195476 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.195409 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" event={"ID":"7fc3c304-52a0-4f96-8f6b-021c866e3f0a","Type":"ContainerStarted","Data":"e52fa04782320c93ee8f3bc1b5d5f2e03da912523cfcc134a3d4d452bc7e5d0f"} Oct 30 16:42:49.234869 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.234809 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podStartSLOduration=1.234795117 podStartE2EDuration="1.234795117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:49.23158173 +0000 UTC m=+1278.381862720" watchObservedRunningTime="2026-10-30 16:42:49.234795117 +0000 UTC m=+1278.385076104" Oct 30 16:42:49.273111 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.273055 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podStartSLOduration=1.273040326 podStartE2EDuration="1.273040326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:49.27115751 +0000 UTC m=+1278.421438498" watchObservedRunningTime="2026-10-30 16:42:49.273040326 +0000 UTC m=+1278.423321313" Oct 30 16:42:49.314231 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.314175 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podStartSLOduration=1.314158822 podStartE2EDuration="1.314158822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:49.312158382 +0000 UTC m=+1278.462439372" watchObservedRunningTime="2026-10-30 16:42:49.314158822 +0000 UTC m=+1278.464439810" Oct 30 16:42:49.387349 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.387318 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:49.388486 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.388462 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4cwj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:42:49.388596 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.388509 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:42:49.406671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.406640 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:49.408181 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.408152 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jfk8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:42:49.408309 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.408210 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:42:49.498297 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.496978 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:42:49.498764 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.498735 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:42:49.499273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:49.498961 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 16:42:50.391533 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:50.391066 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4cwj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:42:50.391533 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:50.391132 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:42:50.407456 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:50.407422 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jfk8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:42:50.407691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:50.407566 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:42:50.505646 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:50.501142 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:42:50.505646 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:50.501202 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 16:42:51.398626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:51.395305 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4cwj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:42:51.398626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:51.395389 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:42:51.416626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:51.414330 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jfk8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:42:51.416626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:51.414418 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:42:51.503787 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:51.503745 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:42:51.504095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:51.504057 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 16:42:52.387642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:52.387201 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4cwj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:42:52.387642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:52.387283 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:42:52.407273 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:52.407233 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jfk8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:42:52.407731 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:52.407282 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:42:52.495660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:52.495588 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:42:52.495839 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:52.495676 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 16:42:53.391481 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:53.391448 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4cwj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:42:53.391715 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:53.391667 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:42:53.407349 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:53.407306 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jfk8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:42:53.407790 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:53.407370 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:42:53.495252 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:53.495214 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:42:53.495427 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:53.495264 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 16:42:53.964662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:53.964332 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9"] Oct 30 16:42:53.994173 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:53.991769 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:53.996283 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:53.996245 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9"] Oct 30 16:42:54.006423 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.005274 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-e4aca49c-serving-cert\"" Oct 30 16:42:54.098000 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.097960 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-maas-api-tls\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.098167 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.098058 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg69z\" (UniqueName: \"kubernetes.io/projected/8cb51b6a-7dba-452f-8fea-f90247d603fb-kube-api-access-pg69z\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.098167 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.098123 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-metrics-certs\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.098167 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.098140 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb51b6a-7dba-452f-8fea-f90247d603fb-openshift-service-ca\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.202760 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.202720 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-maas-api-tls\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.203354 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.203319 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg69z\" (UniqueName: \"kubernetes.io/projected/8cb51b6a-7dba-452f-8fea-f90247d603fb-kube-api-access-pg69z\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.203506 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.203450 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-metrics-certs\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.203506 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.203478 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb51b6a-7dba-452f-8fea-f90247d603fb-openshift-service-ca\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.204577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.204544 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb51b6a-7dba-452f-8fea-f90247d603fb-openshift-service-ca\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.209042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.209019 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-maas-api-tls\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.283552 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.283479 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg69z\" (UniqueName: \"kubernetes.io/projected/8cb51b6a-7dba-452f-8fea-f90247d603fb-kube-api-access-pg69z\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.295642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.288150 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-metrics-certs\") pod \"maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.310564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.309849 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:54.396662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.389011 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4cwj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:42:54.396662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.389067 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:42:54.407307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.407142 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jfk8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:42:54.407307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.407198 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:42:54.497164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.495028 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:42:54.497164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:54.495090 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 16:42:54.841564 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:42:54.840831 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:42:55.322630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:55.318617 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9"] Oct 30 16:42:55.390057 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:55.389994 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4cwj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:42:55.390356 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:55.390322 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:42:55.442628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:55.430008 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jfk8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:42:55.442628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:55.430076 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:42:55.524625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:55.523903 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:42:55.524625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:55.523997 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 16:42:55.581730 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:55.563051 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" event={"ID":"8cb51b6a-7dba-452f-8fea-f90247d603fb","Type":"ContainerStarted","Data":"5dc95c6b22d5191c587e7cd87f3219fc79ca86fbbaf0f0ac7196b03cbf8e2580"} Oct 30 16:42:56.387467 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:56.386969 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4cwj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:42:56.387467 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:56.387032 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:42:56.407246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:56.407124 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jfk8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:42:56.407246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:56.407181 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:42:56.513625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:56.503735 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:42:56.513625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:56.503794 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 16:42:56.613340 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:56.613300 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" event={"ID":"8cb51b6a-7dba-452f-8fea-f90247d603fb","Type":"ContainerStarted","Data":"1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac"} Oct 30 16:42:56.623511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:56.623479 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:42:56.686792 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:56.679854 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" podStartSLOduration=3.193526346 podStartE2EDuration="3.6798347s" totalImagesPullingTime="486.308354ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:42:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:42:56.678462791 +0000 UTC m=+1285.828743780" watchObservedRunningTime="2026-10-30 16:42:56.6798347 +0000 UTC m=+1285.830115692" Oct 30 16:42:57.387703 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:57.387661 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4cwj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" start-of-body= Oct 30 16:42:57.387905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:57.387735 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.90:15021/healthz/ready\": dial tcp 10.134.0.90:15021: connect: connection refused" Oct 30 16:42:57.407316 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:57.407283 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-jfk8q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body= Oct 30 16:42:57.407494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:57.407350 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" Oct 30 16:42:57.495868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:57.495831 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 30 16:42:57.496077 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:57.495898 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 30 16:42:59.052455 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:59.052373 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:59.054034 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:59.054008 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:59.054903 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:59.054883 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:42:59.069016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:59.068970 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:59.070586 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:59.070566 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:59.072261 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:59.072238 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:42:59.496181 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:59.496134 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:42:59.496390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:42:59.496210 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:43:00.550627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:00.544767 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:43:00.550627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:00.544865 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:43:01.293627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:01.292048 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw"] Oct 30 16:43:01.293627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:01.292671 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" podUID="464d29f0-a0f7-4644-925c-bee309ecf68e" containerName="maas-api" containerID="cri-o://ee658c425d3d46fb464d7c4094964614cec0cf264b967b9d5108a691ad8b9889" gracePeriod=30 Oct 30 16:43:01.590777 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:01.589786 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-gjgfk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:43:01.590777 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:01.589892 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:43:01.869073 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:01.868985 6400 generic.go:356] "Generic (PLEG): container finished" podID="464d29f0-a0f7-4644-925c-bee309ecf68e" containerID="ee658c425d3d46fb464d7c4094964614cec0cf264b967b9d5108a691ad8b9889" exitCode=0 Oct 30 16:43:01.869546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:01.869518 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" event={"ID":"464d29f0-a0f7-4644-925c-bee309ecf68e","Type":"ContainerDied","Data":"ee658c425d3d46fb464d7c4094964614cec0cf264b967b9d5108a691ad8b9889"} Oct 30 16:43:02.196690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.187536 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:43:02.196690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.193089 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:43:02.202244 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.197701 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 16:43:02.433625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.432240 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:43:02.511765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.511716 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/464d29f0-a0f7-4644-925c-bee309ecf68e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464d29f0-a0f7-4644-925c-bee309ecf68e-openshift-service-ca\") pod \"464d29f0-a0f7-4644-925c-bee309ecf68e\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " Oct 30 16:43:02.511962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.511880 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-metrics-certs\") pod \"464d29f0-a0f7-4644-925c-bee309ecf68e\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " Oct 30 16:43:02.512122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.512103 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls\") pod \"464d29f0-a0f7-4644-925c-bee309ecf68e\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " Oct 30 16:43:02.512256 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.512238 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/464d29f0-a0f7-4644-925c-bee309ecf68e-kube-api-access-k7bm2\" (UniqueName: \"kubernetes.io/projected/464d29f0-a0f7-4644-925c-bee309ecf68e-kube-api-access-k7bm2\") pod \"464d29f0-a0f7-4644-925c-bee309ecf68e\" (UID: \"464d29f0-a0f7-4644-925c-bee309ecf68e\") " Oct 30 16:43:02.513593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.513501 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464d29f0-a0f7-4644-925c-bee309ecf68e-openshift-service-ca" pod "464d29f0-a0f7-4644-925c-bee309ecf68e" (UID: "464d29f0-a0f7-4644-925c-bee309ecf68e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:02.517936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.517904 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls" pod "464d29f0-a0f7-4644-925c-bee309ecf68e" (UID: "464d29f0-a0f7-4644-925c-bee309ecf68e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:02.517936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.517927 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464d29f0-a0f7-4644-925c-bee309ecf68e-kube-api-access-k7bm2" pod "464d29f0-a0f7-4644-925c-bee309ecf68e" (UID: "464d29f0-a0f7-4644-925c-bee309ecf68e"). InnerVolumeSpecName "kube-api-access-k7bm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:02.519869 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.519822 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-metrics-certs" pod "464d29f0-a0f7-4644-925c-bee309ecf68e" (UID: "464d29f0-a0f7-4644-925c-bee309ecf68e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:43:02.619171 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.619135 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:02.619171 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.619171 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7bm2\" (UniqueName: \"kubernetes.io/projected/464d29f0-a0f7-4644-925c-bee309ecf68e-kube-api-access-k7bm2\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:02.619663 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.619187 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/464d29f0-a0f7-4644-925c-bee309ecf68e-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:02.619663 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.619201 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/464d29f0-a0f7-4644-925c-bee309ecf68e-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:02.633058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.633034 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:43:02.877713 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.877628 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" event={"ID":"464d29f0-a0f7-4644-925c-bee309ecf68e","Type":"ContainerDied","Data":"a54f2e7511982a1ca84995a6ca0e0342f72e47df3eb79e6d4eeacb1e79f64b50"} Oct 30 16:43:02.877713 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.877697 6400 scope.go:122] "RemoveContainer" containerID="ee658c425d3d46fb464d7c4094964614cec0cf264b967b9d5108a691ad8b9889" Oct 30 16:43:02.877917 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.877646 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw" Oct 30 16:43:02.927040 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.927005 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw"] Oct 30 16:43:02.932649 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.932596 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a475e0dd-df7bf465c-8phtw"] Oct 30 16:43:02.968582 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:02.968551 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="464d29f0-a0f7-4644-925c-bee309ecf68e" path="/var/lib/kubelet/pods/464d29f0-a0f7-4644-925c-bee309ecf68e/volumes" Oct 30 16:43:21.928526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:21.928126 6400 patch_prober.go:29] interesting pod/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:43:21.928526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:21.928214 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:43:24.624665 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:43:24.624566 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:43:25.146161 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.146128 6400 generic.go:356] "Generic (PLEG): container finished" podID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerID="a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18" exitCode=0 Oct 30 16:43:25.146337 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.146196 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" event={"ID":"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483","Type":"ContainerDied","Data":"a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18"} Oct 30 16:43:25.146762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.146739 6400 scope.go:122] "RemoveContainer" containerID="a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18" Oct 30 16:43:25.148191 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.148173 6400 generic.go:356] "Generic (PLEG): container finished" podID="07713c9b-864c-43db-9a1b-f7aef356f607" containerID="847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513" exitCode=0 Oct 30 16:43:25.148292 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.148199 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" event={"ID":"07713c9b-864c-43db-9a1b-f7aef356f607","Type":"ContainerDied","Data":"847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513"} Oct 30 16:43:25.148447 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.148417 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:43:25.148529 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.148510 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:43:25.148591 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.148573 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:43:25.148673 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.148520 6400 scope.go:122] "RemoveContainer" containerID="847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513" Oct 30 16:43:25.149925 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.149902 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:43:25.149982 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.149962 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:43:25.150020 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:25.149997 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:43:26.163562 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:26.163524 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" event={"ID":"07713c9b-864c-43db-9a1b-f7aef356f607","Type":"ContainerStarted","Data":"643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b"} Oct 30 16:43:26.167484 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:26.167421 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" event={"ID":"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483","Type":"ContainerStarted","Data":"c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b"} Oct 30 16:43:26.979301 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:26.979270 6400 patch_prober.go:29] interesting pod/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:43:26.979575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:26.979548 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 16:43:27.009154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:27.009115 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:43:27.090248 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:27.090216 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:43:27.090716 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:27.090681 6400 patch_prober.go:29] interesting pod/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 16:43:27.090931 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:27.090892 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 16:43:27.977235 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:27.977200 6400 patch_prober.go:29] interesting pod/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:43:27.977819 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:27.977261 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 16:43:28.080402 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:28.080362 6400 patch_prober.go:29] interesting pod/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 16:43:28.080595 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:28.080419 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 16:43:28.977893 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:28.977859 6400 patch_prober.go:29] interesting pod/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:43:28.978307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:28.977944 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 16:43:29.079888 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:29.079855 6400 patch_prober.go:29] interesting pod/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 16:43:29.080064 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:29.079909 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 16:43:29.977256 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:29.977216 6400 patch_prober.go:29] interesting pod/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:43:29.977447 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:29.977266 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 16:43:30.079850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:30.079812 6400 patch_prober.go:29] interesting pod/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 16:43:30.080277 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:30.079866 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 16:43:30.983739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:30.979729 6400 patch_prober.go:29] interesting pod/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" start-of-body= Oct 30 16:43:30.983739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:30.979815 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.89:15021/healthz/ready\": dial tcp 10.134.0.89:15021: connect: connection refused" Oct 30 16:43:31.005516 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:31.005490 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:43:31.079450 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:31.079415 6400 patch_prober.go:29] interesting pod/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 30 16:43:31.079662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:31.079477 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 30 16:43:31.981559 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:31.981527 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:43:31.982477 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:31.982458 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:43:32.085018 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:32.084982 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:43:32.085526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:32.085497 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:43:32.086377 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:32.086359 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:43:39.099399 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:39.099346 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt"] Oct 30 16:43:39.101008 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:39.100749 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" containerID="cri-o://c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b" gracePeriod=30 Oct 30 16:43:44.386185 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.386131 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:43:44.416725 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.416696 6400 generic.go:356] "Generic (PLEG): container finished" podID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerID="c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b" exitCode=0 Oct 30 16:43:44.416884 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.416746 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" Oct 30 16:43:44.416884 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.416778 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" event={"ID":"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483","Type":"ContainerDied","Data":"c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b"} Oct 30 16:43:44.416884 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.416863 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt" event={"ID":"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483","Type":"ContainerDied","Data":"e26e8d83abe469892261cf63dba3784d54d622f20c712073d0af6d6a540afa47"} Oct 30 16:43:44.416884 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.416879 6400 scope.go:122] "RemoveContainer" containerID="c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b" Oct 30 16:43:44.427797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.427780 6400 scope.go:122] "RemoveContainer" containerID="a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18" Oct 30 16:43:44.437856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.437832 6400 scope.go:122] "RemoveContainer" containerID="c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b" Oct 30 16:43:44.438401 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:43:44.438376 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b\": container with ID starting with c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b not found: ID does not exist" containerID="c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b" Oct 30 16:43:44.438541 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.438518 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b"} err="failed to get container status \"c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b\": rpc error: code = NotFound desc = could not find container \"c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b\": container with ID starting with c2f83f39ff2367f9cf7725e40a3c3c1187d544f1ccf1fc67f2be0b55dc514a6b not found: ID does not exist" Oct 30 16:43:44.438746 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.438544 6400 scope.go:122] "RemoveContainer" containerID="a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18" Oct 30 16:43:44.438873 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:43:44.438849 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18\": container with ID starting with a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18 not found: ID does not exist" containerID="a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18" Oct 30 16:43:44.439002 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.438881 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18"} err="failed to get container status \"a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18\": rpc error: code = NotFound desc = could not find container \"a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18\": container with ID starting with a50b7f500de6b7d2c07054b4fc0bef90407a1400257579dc1d0d3bf49d251a18 not found: ID does not exist" Oct 30 16:43:44.526343 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526313 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-certs\") pod \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " Oct 30 16:43:44.526526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526405 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istiod-ca-cert\") pod \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " Oct 30 16:43:44.526526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526445 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-podinfo\") pod \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " Oct 30 16:43:44.526526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526480 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-socket\") pod \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " Oct 30 16:43:44.526526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526520 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-envoy\") pod \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " Oct 30 16:43:44.526725 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526629 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-certs" pod "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" (UID: "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:44.526725 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526638 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-token\" (UniqueName: \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-token\") pod \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " Oct 30 16:43:44.526827 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526777 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-data\") pod \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " Oct 30 16:43:44.526882 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526864 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-credential-socket\") pod \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " Oct 30 16:43:44.526942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526870 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-socket" pod "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" (UID: "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:44.526942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526888 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istiod-ca-cert" pod "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" (UID: "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:43:44.527047 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.526941 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-kube-api-access-r7b2s\" (UniqueName: \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-kube-api-access-r7b2s\") pod \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\" (UID: \"c07ad2d1-4d6f-491b-95b6-a5b9f9a83483\") " Oct 30 16:43:44.527047 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.527024 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-data" pod "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" (UID: "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:44.527248 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.527216 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-credential-socket" pod "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" (UID: "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:44.527925 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.527900 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:44.528042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.527930 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:44.528042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.527945 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:44.528042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.527960 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:44.528042 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.527974 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:44.529229 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.529194 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-envoy" pod "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" (UID: "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:43:44.529354 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.529321 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-token" pod "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" (UID: "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:44.532971 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.532952 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-podinfo" pod "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" (UID: "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:43:44.533101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.533079 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-kube-api-access-r7b2s" pod "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" (UID: "c07ad2d1-4d6f-491b-95b6-a5b9f9a83483"). InnerVolumeSpecName "kube-api-access-r7b2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:43:44.629951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.629923 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7b2s\" (UniqueName: \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-kube-api-access-r7b2s\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:44.629951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.629949 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:44.629951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.629960 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:44.630201 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.629969 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:43:44.764147 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.764109 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt"] Oct 30 16:43:44.773457 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.773431 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-a0777a0c-openshift-default-5df4d46f9c-7cnwt"] Oct 30 16:43:44.968575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:44.968540 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" path="/var/lib/kubelet/pods/c07ad2d1-4d6f-491b-95b6-a5b9f9a83483/volumes" Oct 30 16:43:53.965963 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.965928 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q"] Oct 30 16:43:53.967360 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.967333 6400 cpu_manager.go:414] "containerMap: removing container" podUID="464d29f0-a0f7-4644-925c-bee309ecf68e" containerName="maas-api" Oct 30 16:43:53.967545 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.967530 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="464d29f0-a0f7-4644-925c-bee309ecf68e" containerName="maas-api" Oct 30 16:43:53.967854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.967838 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" Oct 30 16:43:53.967952 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.967940 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" Oct 30 16:43:53.968049 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.968039 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" Oct 30 16:43:53.968122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.968112 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" Oct 30 16:43:53.968427 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.968412 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="464d29f0-a0f7-4644-925c-bee309ecf68e" containerName="maas-api" Oct 30 16:43:53.968543 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.968532 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" Oct 30 16:43:53.968651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.968640 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c07ad2d1-4d6f-491b-95b6-a5b9f9a83483" containerName="istio-proxy" Oct 30 16:43:53.972161 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.972128 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:53.992187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.992160 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q"] Oct 30 16:43:53.994513 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.994238 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-envoy\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:53.994513 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.994294 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-podinfo\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:53.994513 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.994438 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c3c768-317f-4692-8e4e-27fe738d96cc-istiod-ca-cert\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:53.994513 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.994510 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-certs\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:53.994824 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.994546 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-credential-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:53.994824 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.994760 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:53.994928 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.994844 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-data\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:53.994928 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.994882 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6zn4\" (UniqueName: \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-kube-api-access-m6zn4\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:53.994928 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.994918 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-token\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:53.997628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:53.997591 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-830ba9-openshift-default-dockercfg-jp6bn\"" Oct 30 16:43:54.096653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.096584 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-envoy\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.096858 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.096674 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-podinfo\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.096858 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.096832 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c3c768-317f-4692-8e4e-27fe738d96cc-istiod-ca-cert\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.096991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.096912 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-certs\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.096991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.096947 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-credential-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.097101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.097014 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.097354 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.097327 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-credential-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.097573 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.097548 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-socket\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.097694 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.097669 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-data\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.097758 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.097725 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6zn4\" (UniqueName: \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-kube-api-access-m6zn4\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.097818 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.097768 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-token\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.097968 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.097948 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-certs\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.098153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.098132 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c3c768-317f-4692-8e4e-27fe738d96cc-istiod-ca-cert\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.098411 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.098387 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-data\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.099166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.099142 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-envoy\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.099428 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.099411 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-podinfo\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.123759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.123732 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-token\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.124074 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.124049 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6zn4\" (UniqueName: \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-kube-api-access-m6zn4\") pod \"e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.295628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.293891 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:54.608695 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:43:54.607743 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:43:54.665263 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:43:54.665226 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c3c768_317f_4692_8e4e_27fe738d96cc.slice/crio-15059e106c4beb8719721980e7f6b7dc22c9878b67ac17818f6bf7ed2b0618cf WatchSource:0}: Error finding container 15059e106c4beb8719721980e7f6b7dc22c9878b67ac17818f6bf7ed2b0618cf: Status 404 returned error can't find the container with id 15059e106c4beb8719721980e7f6b7dc22c9878b67ac17818f6bf7ed2b0618cf Oct 30 16:43:54.666817 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.666789 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q"] Oct 30 16:43:54.668370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.668329 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:43:54.668449 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.668419 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:43:54.668511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:54.668467 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:43:55.513590 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:55.513555 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" event={"ID":"19c3c768-317f-4692-8e4e-27fe738d96cc","Type":"ContainerStarted","Data":"01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a"} Oct 30 16:43:55.514010 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:55.513643 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" event={"ID":"19c3c768-317f-4692-8e4e-27fe738d96cc","Type":"ContainerStarted","Data":"15059e106c4beb8719721980e7f6b7dc22c9878b67ac17818f6bf7ed2b0618cf"} Oct 30 16:43:55.569535 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:55.569479 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" podStartSLOduration=2.569464743 podStartE2EDuration="2.569464743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:43:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:43:55.565835713 +0000 UTC m=+1344.716116701" watchObservedRunningTime="2026-10-30 16:43:55.569464743 +0000 UTC m=+1344.719745725" Oct 30 16:43:56.295135 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:56.295087 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:56.300936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:56.300911 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:56.523584 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:56.523554 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:43:56.524636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:43:56.524574 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:44:07.604698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.604652 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx"] Oct 30 16:44:07.609541 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.609506 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.633507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.633479 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-e43400-serving-cert\"" Oct 30 16:44:07.641693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.641658 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx"] Oct 30 16:44:07.645499 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.645473 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f6c435-d09d-4327-9068-3f4ab0cb41f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.645698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.645675 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.645794 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.645779 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xf99h\" (UniqueName: \"kubernetes.io/projected/74f6c435-d09d-4327-9068-3f4ab0cb41f2-kube-api-access-xf99h\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.645886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.645863 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.747085 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.747039 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xf99h\" (UniqueName: \"kubernetes.io/projected/74f6c435-d09d-4327-9068-3f4ab0cb41f2-kube-api-access-xf99h\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.747312 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.747102 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.747312 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.747280 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f6c435-d09d-4327-9068-3f4ab0cb41f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.747454 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.747370 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.748058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.747978 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f6c435-d09d-4327-9068-3f4ab0cb41f2-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.749884 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.749864 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.749960 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.749893 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.769029 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.768992 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xf99h\" (UniqueName: \"kubernetes.io/projected/74f6c435-d09d-4327-9068-3f4ab0cb41f2-kube-api-access-xf99h\") pod \"maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:07.933040 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:07.932953 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:08.246928 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:08.246900 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx"] Oct 30 16:44:08.247206 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:44:08.247186 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f6c435_d09d_4327_9068_3f4ab0cb41f2.slice/crio-5609248898f34c99c98e621b9564e8fb73c094374150491943dfeffb5419c955 WatchSource:0}: Error finding container 5609248898f34c99c98e621b9564e8fb73c094374150491943dfeffb5419c955: Status 404 returned error can't find the container with id 5609248898f34c99c98e621b9564e8fb73c094374150491943dfeffb5419c955 Oct 30 16:44:08.620779 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:08.620671 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" event={"ID":"74f6c435-d09d-4327-9068-3f4ab0cb41f2","Type":"ContainerStarted","Data":"5609248898f34c99c98e621b9564e8fb73c094374150491943dfeffb5419c955"} Oct 30 16:44:09.646618 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:09.646564 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" event={"ID":"74f6c435-d09d-4327-9068-3f4ab0cb41f2","Type":"ContainerStarted","Data":"afaec0b8b4e6cc5e66705cce7629355136525b79b5f1218b49787bafc119dc93"} Oct 30 16:44:09.647189 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:09.646720 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:09.688593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:09.688525 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" podStartSLOduration=2.1753923410000002 podStartE2EDuration="2.68850379s" totalImagesPullingTime="513.111449ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:09.685080434 +0000 UTC m=+1358.835361424" watchObservedRunningTime="2026-10-30 16:44:09.68850379 +0000 UTC m=+1358.838784779" Oct 30 16:44:15.661330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:15.661296 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 16:44:24.604294 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:44:24.604252 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:44:39.957800 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:39.957768 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj"] Oct 30 16:44:39.963661 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:39.963635 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:39.986323 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:39.983039 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w6-e5bfb0-openshift-default-dockercfg-rkdlj\"" Oct 30 16:44:40.000011 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:39.999979 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj"] Oct 30 16:44:40.114627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.113648 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-certs\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.114627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.113727 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8t2t\" (UniqueName: \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-kube-api-access-g8t2t\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.114627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.113760 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istiod-ca-cert\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.114627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.113840 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-socket\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.114627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.113878 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-data\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.114627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.114002 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-podinfo\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.114627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.114145 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-credential-socket\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.114627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.114171 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-envoy\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.114627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.114279 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-token\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.215627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.215434 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-podinfo\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.215996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.215967 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-credential-socket\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.217334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.216117 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-envoy\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.217334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.216228 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-token\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.217334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.216374 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-certs\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.217334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.216407 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8t2t\" (UniqueName: \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-kube-api-access-g8t2t\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.217334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.216437 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istiod-ca-cert\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.217334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.216498 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-socket\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.217334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.216533 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-data\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.225942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.218177 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-certs\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.226821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.222963 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-credential-socket\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.227021 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.226762 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-socket\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.227125 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.226938 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-data\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.227217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.227037 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istiod-ca-cert\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.247375 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.247336 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-envoy\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.249247 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.248096 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-podinfo\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.252827 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.252760 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-token\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.256500 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.256477 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8t2t\" (UniqueName: \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-kube-api-access-g8t2t\") pod \"e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.299325 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.299290 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:40.650345 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:44:40.650306 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4a7872_45cb_498b_a8a4_92eeef9c76b7.slice/crio-9a6b5ca5f9858073b4fe9c24cc837a02a881ec96f4dc45b2e9cf8e06a42f3b5b WatchSource:0}: Error finding container 9a6b5ca5f9858073b4fe9c24cc837a02a881ec96f4dc45b2e9cf8e06a42f3b5b: Status 404 returned error can't find the container with id 9a6b5ca5f9858073b4fe9c24cc837a02a881ec96f4dc45b2e9cf8e06a42f3b5b Oct 30 16:44:40.651548 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.651522 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj"] Oct 30 16:44:40.653348 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.653310 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:40.653417 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.653397 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:40.653454 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:40.653438 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:41.014641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:41.014584 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" event={"ID":"fc4a7872-45cb-498b-a8a4-92eeef9c76b7","Type":"ContainerStarted","Data":"2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393"} Oct 30 16:44:41.015202 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:41.014657 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" event={"ID":"fc4a7872-45cb-498b-a8a4-92eeef9c76b7","Type":"ContainerStarted","Data":"9a6b5ca5f9858073b4fe9c24cc837a02a881ec96f4dc45b2e9cf8e06a42f3b5b"} Oct 30 16:44:41.057864 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:41.057810 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" podStartSLOduration=2.057795505 podStartE2EDuration="2.057795505s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:41.055806453 +0000 UTC m=+1390.206087444" watchObservedRunningTime="2026-10-30 16:44:41.057795505 +0000 UTC m=+1390.208076492" Oct 30 16:44:41.300057 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:41.299970 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:41.306996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:41.306967 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:42.037117 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:42.037090 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:42.038173 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:42.038154 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:44:49.469655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.469578 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh"] Oct 30 16:44:49.477355 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.477333 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.496647 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.493132 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-05810b-openshift-default-dockercfg-cf5d9\"" Oct 30 16:44:49.507779 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.507753 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh"] Oct 30 16:44:49.612627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.607671 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.612627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.607747 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.612627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.607818 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.612627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.607843 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.612627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.607900 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcac11b-bb6f-4004-8830-e52f952f08a5-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.612627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.607974 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.612627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.608048 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.612627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.608183 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.612627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.608294 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mr8tx\" (UniqueName: \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-kube-api-access-mr8tx\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.724118 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.724026 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.724352 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.724334 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.724493 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.724480 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.724591 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.724576 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.724757 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.724740 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcac11b-bb6f-4004-8830-e52f952f08a5-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.724921 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.724905 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.725065 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.725051 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.725257 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.725243 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.725399 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.725385 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mr8tx\" (UniqueName: \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-kube-api-access-mr8tx\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.737192 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.729515 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcac11b-bb6f-4004-8830-e52f952f08a5-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.768696 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.735821 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.768937 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.736575 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.769036 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.737674 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.776694 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.737952 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.803015 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.802979 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mr8tx\" (UniqueName: \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-kube-api-access-mr8tx\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.807523 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.807461 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.923837 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.923748 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:49.960493 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:49.960418 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:50.097987 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:50.097952 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:50.730076 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:44:50.730024 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fcac11b_bb6f_4004_8830_e52f952f08a5.slice/crio-38c8aa1d32b59a50be2be7acb57ed8a0d54de93f000e973a645405a9bf371695 WatchSource:0}: Error finding container 38c8aa1d32b59a50be2be7acb57ed8a0d54de93f000e973a645405a9bf371695: Status 404 returned error can't find the container with id 38c8aa1d32b59a50be2be7acb57ed8a0d54de93f000e973a645405a9bf371695 Oct 30 16:44:50.771953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:50.739745 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:50.771953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:50.763823 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:50.771953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:50.763876 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:50.816879 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:50.816847 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh"] Oct 30 16:44:51.105863 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.105822 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" event={"ID":"6fcac11b-bb6f-4004-8830-e52f952f08a5","Type":"ContainerStarted","Data":"9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b"} Oct 30 16:44:51.106035 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.105905 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" event={"ID":"6fcac11b-bb6f-4004-8830-e52f952f08a5","Type":"ContainerStarted","Data":"38c8aa1d32b59a50be2be7acb57ed8a0d54de93f000e973a645405a9bf371695"} Oct 30 16:44:51.180870 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.180805 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" podStartSLOduration=2.180771649 podStartE2EDuration="2.180771649s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:51.16782155 +0000 UTC m=+1400.318102538" watchObservedRunningTime="2026-10-30 16:44:51.180771649 +0000 UTC m=+1400.331052637" Oct 30 16:44:51.191868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.191828 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm"] Oct 30 16:44:51.195395 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.195376 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.227723 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.227650 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-dc74d5-openshift-default-dockercfg-7dmmg\"" Oct 30 16:44:51.239688 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.239655 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm"] Oct 30 16:44:51.345006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.334546 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdldf\" (UniqueName: \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-kube-api-access-hdldf\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.345006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.334659 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-envoy\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.345006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.334787 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-podinfo\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.345006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.334884 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-token\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.345006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.334911 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a964e5-301d-4f97-b692-f2c9797276c4-istiod-ca-cert\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.345006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.335036 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-data\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.345006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.335068 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.345006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.335122 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-certs\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.345006 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.335209 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-credential-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.438627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.437569 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-credential-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.438627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.437693 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdldf\" (UniqueName: \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-kube-api-access-hdldf\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.438627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.437769 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-envoy\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.438627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.437873 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-podinfo\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.438627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.437974 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-token\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.438627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.438000 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a964e5-301d-4f97-b692-f2c9797276c4-istiod-ca-cert\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.438627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.438110 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-data\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.438627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.438138 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.438627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.438191 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-certs\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.476520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.453592 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-credential-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.476520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.453672 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-certs\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.476520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.454309 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a964e5-301d-4f97-b692-f2c9797276c4-istiod-ca-cert\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.476520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.455210 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-data\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.476520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.455640 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-socket\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.476520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.467017 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-envoy\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.486782 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.485290 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-podinfo\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.498122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.494028 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-token\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.498122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.494937 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdldf\" (UniqueName: \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-kube-api-access-hdldf\") pod \"e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.554661 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.535123 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:51.996021 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:51.995985 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm"] Oct 30 16:44:51.997459 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:44:51.997429 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a964e5_301d_4f97_b692_f2c9797276c4.slice/crio-e98619113f239b9a761cc12425086395c4acba7cf896fa5c8b2ab06b836c6071 WatchSource:0}: Error finding container e98619113f239b9a761cc12425086395c4acba7cf896fa5c8b2ab06b836c6071: Status 404 returned error can't find the container with id e98619113f239b9a761cc12425086395c4acba7cf896fa5c8b2ab06b836c6071 Oct 30 16:44:52.000686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.000648 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:52.000801 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.000757 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:52.000859 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.000798 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:52.098726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.098701 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:52.104776 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.104754 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:52.140677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.129915 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" event={"ID":"a0a964e5-301d-4f97-b692-f2c9797276c4","Type":"ContainerStarted","Data":"300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb"} Oct 30 16:44:52.140677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.129975 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" event={"ID":"a0a964e5-301d-4f97-b692-f2c9797276c4","Type":"ContainerStarted","Data":"e98619113f239b9a761cc12425086395c4acba7cf896fa5c8b2ab06b836c6071"} Oct 30 16:44:52.140677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.130005 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:52.140677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.135999 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:44:52.200643 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.200568 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podStartSLOduration=1.200551489 podStartE2EDuration="1.200551489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:52.198976753 +0000 UTC m=+1401.349257739" watchObservedRunningTime="2026-10-30 16:44:52.200551489 +0000 UTC m=+1401.350832478" Oct 30 16:44:52.538741 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.538705 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:52.980587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.980504 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg"] Oct 30 16:44:52.985182 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:52.985154 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.002466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.002413 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-1b7bb3-openshift-default-dockercfg-gqvnj\"" Oct 30 16:44:53.003087 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.003057 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg"] Oct 30 16:44:53.127626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.127322 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.127626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.127401 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7623e9c-0154-4059-bf52-027533652d9e-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.127626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.127508 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.127626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.127596 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr94s\" (UniqueName: \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-kube-api-access-vr94s\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.128011 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.127653 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.128011 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.127693 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7623e9c-0154-4059-bf52-027533652d9e-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.140628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.139038 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.140628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.139162 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.140628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.139276 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.244574 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.244683 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.244774 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.244834 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7623e9c-0154-4059-bf52-027533652d9e-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.244913 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.245009 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr94s\" (UniqueName: \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-kube-api-access-vr94s\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.245043 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.245069 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7623e9c-0154-4059-bf52-027533652d9e-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.245245 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.245994 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-credential-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.246233 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-socket\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.246792 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-certs\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.248909 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7623e9c-0154-4059-bf52-027533652d9e-istiod-ca-cert\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.249826 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-data\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.263951 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-envoy\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.279634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.264456 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7623e9c-0154-4059-bf52-027533652d9e-istio-podinfo\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.280549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.279795 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-istio-token\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.293645 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.288244 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr94s\" (UniqueName: \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-kube-api-access-vr94s\") pod \"e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.320628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.319912 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:53.555737 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.555403 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:53.869264 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.869237 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg"] Oct 30 16:44:53.871020 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:44:53.870986 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7623e9c_0154_4059_bf52_027533652d9e.slice/crio-f0d8f18625e7e8acef03b0d869a942e98e43a2410ef9b0e72473e48800dad00c WatchSource:0}: Error finding container f0d8f18625e7e8acef03b0d869a942e98e43a2410ef9b0e72473e48800dad00c: Status 404 returned error can't find the container with id f0d8f18625e7e8acef03b0d869a942e98e43a2410ef9b0e72473e48800dad00c Oct 30 16:44:53.874933 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.874892 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:53.875044 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.874967 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:53.875044 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:53.875002 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:44:54.156927 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.155472 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" event={"ID":"e7623e9c-0154-4059-bf52-027533652d9e","Type":"ContainerStarted","Data":"40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47"} Oct 30 16:44:54.156927 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.155529 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" event={"ID":"e7623e9c-0154-4059-bf52-027533652d9e","Type":"ContainerStarted","Data":"f0d8f18625e7e8acef03b0d869a942e98e43a2410ef9b0e72473e48800dad00c"} Oct 30 16:44:54.156927 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.156540 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:54.157889 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.157838 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:44:54.200144 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.200078 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" podStartSLOduration=2.200062149 podStartE2EDuration="2.200062149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:54.196889745 +0000 UTC m=+1403.347170733" watchObservedRunningTime="2026-10-30 16:44:54.200062149 +0000 UTC m=+1403.350343137" Oct 30 16:44:54.346844 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.346758 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:54.347738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.347387 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:44:54.347738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.347432 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:44:54.622628 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:44:54.621802 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:44:54.662770 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.662740 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf"] Oct 30 16:44:54.666500 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.666476 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.679124 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.679095 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-b8af660d-serving-cert\"" Oct 30 16:44:54.693216 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.693171 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf"] Oct 30 16:44:54.799654 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.799619 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-maas-api-tls\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.799855 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.799731 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-openshift-service-ca\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.799855 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.799793 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqs77\" (UniqueName: \"kubernetes.io/projected/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-kube-api-access-sqs77\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.799855 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.799849 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-metrics-certs\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.902098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.901993 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-maas-api-tls\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.902098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.902038 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-openshift-service-ca\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.902098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.902060 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqs77\" (UniqueName: \"kubernetes.io/projected/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-kube-api-access-sqs77\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.902501 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.902106 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-metrics-certs\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.902501 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:44:54.902148 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-serving-cert: secret "maas-api-e2e-mt-b8af660d-serving-cert" not found Oct 30 16:44:54.902501 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:44:54.902219 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-maas-api-tls podName:58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb nodeName:}" failed. No retries permitted until 2026-10-30 16:44:55.402199562 +0000 UTC m=+1404.552480545 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-maas-api-tls") pod "maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" (UID: "58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb") : secret "maas-api-e2e-mt-b8af660d-serving-cert" not found Oct 30 16:44:54.902815 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.902794 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-openshift-service-ca\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.905057 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.905029 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-metrics-certs\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:54.928076 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:54.928046 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqs77\" (UniqueName: \"kubernetes.io/projected/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-kube-api-access-sqs77\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:55.326494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:55.326462 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:55.422862 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:55.422830 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-maas-api-tls\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:55.425483 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:55.425459 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-maas-api-tls\") pod \"maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:55.583844 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:55.583755 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:55.921384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:55.921293 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf"] Oct 30 16:44:56.499333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:56.499293 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" event={"ID":"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb","Type":"ContainerStarted","Data":"b87864f0f3d6bf3f5b085af645247182bf2860b74c981ced08d9b3d8df9f7dab"} Oct 30 16:44:56.500842 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:56.500762 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:56.502338 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:56.502311 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:44:57.509932 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:57.509886 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" event={"ID":"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb","Type":"ContainerStarted","Data":"c1f6e0cf5ad63907957042d651ea0b363f89a4df9c0739578a4b00a540fa96ab"} Oct 30 16:44:57.510411 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:57.509984 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:44:57.554630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:57.554544 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" podStartSLOduration=2.982314579 podStartE2EDuration="3.554524418s" totalImagesPullingTime="572.209839ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:44:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:44:57.553242444 +0000 UTC m=+1406.703523427" watchObservedRunningTime="2026-10-30 16:44:57.554524418 +0000 UTC m=+1406.704805407" Oct 30 16:44:57.564810 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:57.564762 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:44:57.564966 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:44:57.564855 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:00.195526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.195489 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s"] Oct 30 16:45:00.201400 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.201365 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9"] Oct 30 16:45:00.204388 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.204360 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" Oct 30 16:45:00.205785 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.205766 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc"] Oct 30 16:45:00.205957 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.205936 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" Oct 30 16:45:00.209480 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.209459 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" Oct 30 16:45:00.213560 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.213538 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm"] Oct 30 16:45:00.216997 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.216978 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" Oct 30 16:45:00.218617 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.218582 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 16:45:00.225857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.225832 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s"] Oct 30 16:45:00.234227 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.231683 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc"] Oct 30 16:45:00.247289 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.246807 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm"] Oct 30 16:45:00.266108 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.263916 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9"] Oct 30 16:45:00.296505 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.296419 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsk9b\" (UniqueName: \"kubernetes.io/projected/7aecd31a-5a84-4228-8511-4449e12b6ea3-kube-api-access-rsk9b\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc\" (UID: \"7aecd31a-5a84-4228-8511-4449e12b6ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" Oct 30 16:45:00.296693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.296536 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4qsb\" (UniqueName: \"kubernetes.io/projected/31d5de4a-8186-4866-800a-a78d45252020-kube-api-access-v4qsb\") pod \"maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s\" (UID: \"31d5de4a-8186-4866-800a-a78d45252020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" Oct 30 16:45:00.296693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.296614 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwd6m\" (UniqueName: \"kubernetes.io/projected/732a9fa1-7151-44cf-a120-e264fddb5a23-kube-api-access-dwd6m\") pod \"maas-api-key-cleanup-29889645-6czvm\" (UID: \"732a9fa1-7151-44cf-a120-e264fddb5a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" Oct 30 16:45:00.296848 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.296782 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxg2k\" (UniqueName: \"kubernetes.io/projected/e900b162-79ab-442e-b0bc-f70844a9165f-kube-api-access-cxg2k\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9\" (UID: \"e900b162-79ab-442e-b0bc-f70844a9165f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" Oct 30 16:45:00.296899 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.296848 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732a9fa1-7151-44cf-a120-e264fddb5a23-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889645-6czvm\" (UID: \"732a9fa1-7151-44cf-a120-e264fddb5a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" Oct 30 16:45:00.296957 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.296936 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e900b162-79ab-442e-b0bc-f70844a9165f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9\" (UID: \"e900b162-79ab-442e-b0bc-f70844a9165f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" Oct 30 16:45:00.297005 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.296994 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d5de4a-8186-4866-800a-a78d45252020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s\" (UID: \"31d5de4a-8186-4866-800a-a78d45252020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" Oct 30 16:45:00.297058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.297045 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aecd31a-5a84-4228-8511-4449e12b6ea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc\" (UID: \"7aecd31a-5a84-4228-8511-4449e12b6ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" Oct 30 16:45:00.398470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.398383 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsk9b\" (UniqueName: \"kubernetes.io/projected/7aecd31a-5a84-4228-8511-4449e12b6ea3-kube-api-access-rsk9b\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc\" (UID: \"7aecd31a-5a84-4228-8511-4449e12b6ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" Oct 30 16:45:00.398694 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.398476 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4qsb\" (UniqueName: \"kubernetes.io/projected/31d5de4a-8186-4866-800a-a78d45252020-kube-api-access-v4qsb\") pod \"maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s\" (UID: \"31d5de4a-8186-4866-800a-a78d45252020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" Oct 30 16:45:00.398694 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.398552 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwd6m\" (UniqueName: \"kubernetes.io/projected/732a9fa1-7151-44cf-a120-e264fddb5a23-kube-api-access-dwd6m\") pod \"maas-api-key-cleanup-29889645-6czvm\" (UID: \"732a9fa1-7151-44cf-a120-e264fddb5a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" Oct 30 16:45:00.398833 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.398709 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxg2k\" (UniqueName: \"kubernetes.io/projected/e900b162-79ab-442e-b0bc-f70844a9165f-kube-api-access-cxg2k\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9\" (UID: \"e900b162-79ab-442e-b0bc-f70844a9165f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" Oct 30 16:45:00.398833 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.398766 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732a9fa1-7151-44cf-a120-e264fddb5a23-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889645-6czvm\" (UID: \"732a9fa1-7151-44cf-a120-e264fddb5a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" Oct 30 16:45:00.398929 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.398847 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e900b162-79ab-442e-b0bc-f70844a9165f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9\" (UID: \"e900b162-79ab-442e-b0bc-f70844a9165f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" Oct 30 16:45:00.398929 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.398893 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d5de4a-8186-4866-800a-a78d45252020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s\" (UID: \"31d5de4a-8186-4866-800a-a78d45252020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" Oct 30 16:45:00.399036 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.398948 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aecd31a-5a84-4228-8511-4449e12b6ea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc\" (UID: \"7aecd31a-5a84-4228-8511-4449e12b6ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" Oct 30 16:45:00.399643 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.399579 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d5de4a-8186-4866-800a-a78d45252020-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s\" (UID: \"31d5de4a-8186-4866-800a-a78d45252020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" Oct 30 16:45:00.399643 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.399595 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e900b162-79ab-442e-b0bc-f70844a9165f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9\" (UID: \"e900b162-79ab-442e-b0bc-f70844a9165f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" Oct 30 16:45:00.399778 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.399704 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aecd31a-5a84-4228-8511-4449e12b6ea3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc\" (UID: \"7aecd31a-5a84-4228-8511-4449e12b6ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" Oct 30 16:45:00.399778 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.399713 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732a9fa1-7151-44cf-a120-e264fddb5a23-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889645-6czvm\" (UID: \"732a9fa1-7151-44cf-a120-e264fddb5a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" Oct 30 16:45:00.420502 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.420468 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4qsb\" (UniqueName: \"kubernetes.io/projected/31d5de4a-8186-4866-800a-a78d45252020-kube-api-access-v4qsb\") pod \"maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s\" (UID: \"31d5de4a-8186-4866-800a-a78d45252020\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" Oct 30 16:45:00.420878 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.420849 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsk9b\" (UniqueName: \"kubernetes.io/projected/7aecd31a-5a84-4228-8511-4449e12b6ea3-kube-api-access-rsk9b\") pod \"maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc\" (UID: \"7aecd31a-5a84-4228-8511-4449e12b6ea3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" Oct 30 16:45:00.420951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.420879 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwd6m\" (UniqueName: \"kubernetes.io/projected/732a9fa1-7151-44cf-a120-e264fddb5a23-kube-api-access-dwd6m\") pod \"maas-api-key-cleanup-29889645-6czvm\" (UID: \"732a9fa1-7151-44cf-a120-e264fddb5a23\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" Oct 30 16:45:00.422101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.422077 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxg2k\" (UniqueName: \"kubernetes.io/projected/e900b162-79ab-442e-b0bc-f70844a9165f-kube-api-access-cxg2k\") pod \"maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9\" (UID: \"e900b162-79ab-442e-b0bc-f70844a9165f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" Oct 30 16:45:00.531656 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.531621 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" Oct 30 16:45:00.537099 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.537071 6400 generic.go:356] "Generic (PLEG): container finished" podID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerID="01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a" exitCode=0 Oct 30 16:45:00.537237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.537150 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" event={"ID":"19c3c768-317f-4692-8e4e-27fe738d96cc","Type":"ContainerDied","Data":"01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a"} Oct 30 16:45:00.537639 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.537583 6400 scope.go:122] "RemoveContainer" containerID="01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a" Oct 30 16:45:00.541956 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.541938 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" Oct 30 16:45:00.542690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.542656 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:00.542787 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.542732 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:00.542787 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.542764 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:00.564356 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.564174 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" Oct 30 16:45:00.576574 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.574752 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" Oct 30 16:45:00.916003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.915596 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s"] Oct 30 16:45:00.934340 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.934223 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9"] Oct 30 16:45:00.954884 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:45:00.954807 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode900b162_79ab_442e_b0bc_f70844a9165f.slice/crio-336d0b8b7119ac0559f3009ef3ea15698ffaef6bf37c47d77a9830b2fd9a40f6 WatchSource:0}: Error finding container 336d0b8b7119ac0559f3009ef3ea15698ffaef6bf37c47d77a9830b2fd9a40f6: Status 404 returned error can't find the container with id 336d0b8b7119ac0559f3009ef3ea15698ffaef6bf37c47d77a9830b2fd9a40f6 Oct 30 16:45:00.989716 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:00.989683 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc"] Oct 30 16:45:01.008020 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.007982 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm"] Oct 30 16:45:01.549940 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.549643 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" event={"ID":"31d5de4a-8186-4866-800a-a78d45252020","Type":"ContainerStarted","Data":"01664e54d1133448c4cf3c3d8fe1aed8d70152f3c594f49154b15c8dfc28d70a"} Oct 30 16:45:01.549940 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.549712 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" event={"ID":"31d5de4a-8186-4866-800a-a78d45252020","Type":"ContainerStarted","Data":"dd2ff8fdc2aecb61eac7ba5e738de56acce36b60ada85bd0c21022193c5f1386"} Oct 30 16:45:01.552334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.552301 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" event={"ID":"19c3c768-317f-4692-8e4e-27fe738d96cc","Type":"ContainerStarted","Data":"b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e"} Oct 30 16:45:01.555467 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.555440 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" event={"ID":"732a9fa1-7151-44cf-a120-e264fddb5a23","Type":"ContainerStarted","Data":"3df8ff623a044f70b443130f7d90c97ac903034f7aee0fdcf35e7066db96c15c"} Oct 30 16:45:01.555623 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.555490 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" event={"ID":"732a9fa1-7151-44cf-a120-e264fddb5a23","Type":"ContainerStarted","Data":"779e39c4d98e730b13adb431058e141b6890e12457313c4b0b1e3bf15c8994b9"} Oct 30 16:45:01.557459 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.557430 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" event={"ID":"e900b162-79ab-442e-b0bc-f70844a9165f","Type":"ContainerStarted","Data":"c92124164a56a6a2dd80c632ca351bd592612583ca25b582501a3f78b690f751"} Oct 30 16:45:01.557577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.557485 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" event={"ID":"e900b162-79ab-442e-b0bc-f70844a9165f","Type":"ContainerStarted","Data":"336d0b8b7119ac0559f3009ef3ea15698ffaef6bf37c47d77a9830b2fd9a40f6"} Oct 30 16:45:01.559574 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.559541 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" event={"ID":"7aecd31a-5a84-4228-8511-4449e12b6ea3","Type":"ContainerStarted","Data":"29f204c660995d483c54b7678c85a7c188e69a92abd6007b6d22438d125f03c8"} Oct 30 16:45:01.559732 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.559591 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" event={"ID":"7aecd31a-5a84-4228-8511-4449e12b6ea3","Type":"ContainerStarted","Data":"bbcaf0795f7f9b570f460608720b2a1d577693c51808cf66b2eea47bb17abbc3"} Oct 30 16:45:01.584894 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.584826 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" podStartSLOduration=1.584805259 podStartE2EDuration="1.584805259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:01.584331311 +0000 UTC m=+1410.734612300" watchObservedRunningTime="2026-10-30 16:45:01.584805259 +0000 UTC m=+1410.735086246" Oct 30 16:45:01.661626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.661323 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" podStartSLOduration=1.661307507 podStartE2EDuration="1.661307507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:01.658032344 +0000 UTC m=+1410.808313333" watchObservedRunningTime="2026-10-30 16:45:01.661307507 +0000 UTC m=+1410.811588495" Oct 30 16:45:01.694463 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.694412 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" podStartSLOduration=1.694392826 podStartE2EDuration="1.694392826s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:01.692919483 +0000 UTC m=+1410.843200472" watchObservedRunningTime="2026-10-30 16:45:01.694392826 +0000 UTC m=+1410.844673817" Oct 30 16:45:01.729891 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:01.729838 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" podStartSLOduration=1.7298237250000001 podStartE2EDuration="1.729823725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:01.727203411 +0000 UTC m=+1410.877484402" watchObservedRunningTime="2026-10-30 16:45:01.729823725 +0000 UTC m=+1410.880104712" Oct 30 16:45:01.993014 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:01.992977 6400 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31d5de4a_8186_4866_800a_a78d45252020.slice/crio-01664e54d1133448c4cf3c3d8fe1aed8d70152f3c594f49154b15c8dfc28d70a.scope\": RecentStats: unable to find data in memory cache]" Oct 30 16:45:02.295101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.295066 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:45:02.295979 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.295943 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:45:02.296050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.296022 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 16:45:02.567982 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.567935 6400 generic.go:356] "Generic (PLEG): container finished" podID="e900b162-79ab-442e-b0bc-f70844a9165f" containerID="c92124164a56a6a2dd80c632ca351bd592612583ca25b582501a3f78b690f751" exitCode=60 Oct 30 16:45:02.568452 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.568058 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" event={"ID":"e900b162-79ab-442e-b0bc-f70844a9165f","Type":"ContainerDied","Data":"c92124164a56a6a2dd80c632ca351bd592612583ca25b582501a3f78b690f751"} Oct 30 16:45:02.568510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.568463 6400 scope.go:122] "RemoveContainer" containerID="c92124164a56a6a2dd80c632ca351bd592612583ca25b582501a3f78b690f751" Oct 30 16:45:02.571514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.571490 6400 generic.go:356] "Generic (PLEG): container finished" podID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerID="29f204c660995d483c54b7678c85a7c188e69a92abd6007b6d22438d125f03c8" exitCode=60 Oct 30 16:45:02.571701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.571672 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" event={"ID":"7aecd31a-5a84-4228-8511-4449e12b6ea3","Type":"ContainerDied","Data":"29f204c660995d483c54b7678c85a7c188e69a92abd6007b6d22438d125f03c8"} Oct 30 16:45:02.571933 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.571902 6400 scope.go:122] "RemoveContainer" containerID="29f204c660995d483c54b7678c85a7c188e69a92abd6007b6d22438d125f03c8" Oct 30 16:45:02.581732 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.581505 6400 generic.go:356] "Generic (PLEG): container finished" podID="31d5de4a-8186-4866-800a-a78d45252020" containerID="01664e54d1133448c4cf3c3d8fe1aed8d70152f3c594f49154b15c8dfc28d70a" exitCode=60 Oct 30 16:45:02.581732 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.581669 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" event={"ID":"31d5de4a-8186-4866-800a-a78d45252020","Type":"ContainerDied","Data":"01664e54d1133448c4cf3c3d8fe1aed8d70152f3c594f49154b15c8dfc28d70a"} Oct 30 16:45:02.582247 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.581971 6400 scope.go:122] "RemoveContainer" containerID="01664e54d1133448c4cf3c3d8fe1aed8d70152f3c594f49154b15c8dfc28d70a" Oct 30 16:45:02.583462 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.583439 6400 generic.go:356] "Generic (PLEG): container finished" podID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerID="3df8ff623a044f70b443130f7d90c97ac903034f7aee0fdcf35e7066db96c15c" exitCode=60 Oct 30 16:45:02.583560 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.583506 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" event={"ID":"732a9fa1-7151-44cf-a120-e264fddb5a23","Type":"ContainerDied","Data":"3df8ff623a044f70b443130f7d90c97ac903034f7aee0fdcf35e7066db96c15c"} Oct 30 16:45:02.583787 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:02.583769 6400 scope.go:122] "RemoveContainer" containerID="3df8ff623a044f70b443130f7d90c97ac903034f7aee0fdcf35e7066db96c15c" Oct 30 16:45:03.043567 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:03.043510 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54"] Oct 30 16:45:03.047954 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:03.047929 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2a1fd15-jxw54"] Oct 30 16:45:03.295335 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:03.295303 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:45:03.295513 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:03.295356 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 16:45:03.541398 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:03.541368 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:45:03.596009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:03.595970 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" event={"ID":"e900b162-79ab-442e-b0bc-f70844a9165f","Type":"ContainerStarted","Data":"121d0f2d77bdf642e7f57264e46bce04c7a5a84124d562ea4f0108f40083d6dd"} Oct 30 16:45:03.599370 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:03.599342 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" event={"ID":"7aecd31a-5a84-4228-8511-4449e12b6ea3","Type":"ContainerStarted","Data":"18b3f060277f07af58281790c92d80c8d61bba4c52d85c4aed5ddc898a7e5007"} Oct 30 16:45:03.602141 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:03.602115 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" event={"ID":"31d5de4a-8186-4866-800a-a78d45252020","Type":"ContainerStarted","Data":"fc8d5a1a0bd59f64a8207cb14d16a003bc33e96dd2cbb73e827b34f8f8f670bf"} Oct 30 16:45:03.604819 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:03.604792 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" event={"ID":"732a9fa1-7151-44cf-a120-e264fddb5a23","Type":"ContainerStarted","Data":"df3348c7a0db1993d0b821e286edb849a98ff5b94645f5686a537d363497207e"} Oct 30 16:45:04.294717 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.294682 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:45:04.294899 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.294739 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 16:45:04.636898 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.636806 6400 generic.go:356] "Generic (PLEG): container finished" podID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerID="18b3f060277f07af58281790c92d80c8d61bba4c52d85c4aed5ddc898a7e5007" exitCode=60 Oct 30 16:45:04.637296 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.636984 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" event={"ID":"7aecd31a-5a84-4228-8511-4449e12b6ea3","Type":"ContainerDied","Data":"18b3f060277f07af58281790c92d80c8d61bba4c52d85c4aed5ddc898a7e5007"} Oct 30 16:45:04.637296 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.637044 6400 scope.go:122] "RemoveContainer" containerID="29f204c660995d483c54b7678c85a7c188e69a92abd6007b6d22438d125f03c8" Oct 30 16:45:04.637658 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.637625 6400 scope.go:122] "RemoveContainer" containerID="18b3f060277f07af58281790c92d80c8d61bba4c52d85c4aed5ddc898a7e5007" Oct 30 16:45:04.638098 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:04.638071 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc_odh-ai-gateway-infra(7aecd31a-5a84-4228-8511-4449e12b6ea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" Oct 30 16:45:04.641860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.641834 6400 generic.go:356] "Generic (PLEG): container finished" podID="31d5de4a-8186-4866-800a-a78d45252020" containerID="fc8d5a1a0bd59f64a8207cb14d16a003bc33e96dd2cbb73e827b34f8f8f670bf" exitCode=60 Oct 30 16:45:04.642098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.642072 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" event={"ID":"31d5de4a-8186-4866-800a-a78d45252020","Type":"ContainerDied","Data":"fc8d5a1a0bd59f64a8207cb14d16a003bc33e96dd2cbb73e827b34f8f8f670bf"} Oct 30 16:45:04.642539 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.642520 6400 scope.go:122] "RemoveContainer" containerID="fc8d5a1a0bd59f64a8207cb14d16a003bc33e96dd2cbb73e827b34f8f8f670bf" Oct 30 16:45:04.643292 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:04.643054 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s_odh-ai-gateway-infra(31d5de4a-8186-4866-800a-a78d45252020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" podUID="31d5de4a-8186-4866-800a-a78d45252020" Oct 30 16:45:04.644476 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.644455 6400 generic.go:356] "Generic (PLEG): container finished" podID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerID="df3348c7a0db1993d0b821e286edb849a98ff5b94645f5686a537d363497207e" exitCode=60 Oct 30 16:45:04.644556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.644484 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" event={"ID":"732a9fa1-7151-44cf-a120-e264fddb5a23","Type":"ContainerDied","Data":"df3348c7a0db1993d0b821e286edb849a98ff5b94645f5686a537d363497207e"} Oct 30 16:45:04.644863 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.644847 6400 scope.go:122] "RemoveContainer" containerID="df3348c7a0db1993d0b821e286edb849a98ff5b94645f5686a537d363497207e" Oct 30 16:45:04.645256 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:04.645236 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889645-6czvm_odh-ai-gateway-infra(732a9fa1-7151-44cf-a120-e264fddb5a23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" Oct 30 16:45:04.646904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.646879 6400 generic.go:356] "Generic (PLEG): container finished" podID="e900b162-79ab-442e-b0bc-f70844a9165f" containerID="121d0f2d77bdf642e7f57264e46bce04c7a5a84124d562ea4f0108f40083d6dd" exitCode=60 Oct 30 16:45:04.647001 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.646959 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" event={"ID":"e900b162-79ab-442e-b0bc-f70844a9165f","Type":"ContainerDied","Data":"121d0f2d77bdf642e7f57264e46bce04c7a5a84124d562ea4f0108f40083d6dd"} Oct 30 16:45:04.647217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.647200 6400 scope.go:122] "RemoveContainer" containerID="121d0f2d77bdf642e7f57264e46bce04c7a5a84124d562ea4f0108f40083d6dd" Oct 30 16:45:04.647525 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:04.647504 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9_odh-ai-gateway-infra(e900b162-79ab-442e-b0bc-f70844a9165f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" Oct 30 16:45:04.652381 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.652366 6400 scope.go:122] "RemoveContainer" containerID="01664e54d1133448c4cf3c3d8fe1aed8d70152f3c594f49154b15c8dfc28d70a" Oct 30 16:45:04.665223 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.664985 6400 scope.go:122] "RemoveContainer" containerID="3df8ff623a044f70b443130f7d90c97ac903034f7aee0fdcf35e7066db96c15c" Oct 30 16:45:04.679222 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.679203 6400 scope.go:122] "RemoveContainer" containerID="c92124164a56a6a2dd80c632ca351bd592612583ca25b582501a3f78b690f751" Oct 30 16:45:04.704482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.704175 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9"] Oct 30 16:45:04.704671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.704488 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" podUID="8cb51b6a-7dba-452f-8fea-f90247d603fb" containerName="maas-api" containerID="cri-o://1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac" gracePeriod=30 Oct 30 16:45:04.789628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:04.789582 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s"] Oct 30 16:45:05.063179 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.063143 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750c619f-fc91-4814-99a9-66b3015350bb" path="/var/lib/kubelet/pods/750c619f-fc91-4814-99a9-66b3015350bb/volumes" Oct 30 16:45:05.355712 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.337217 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:45:05.355712 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.337286 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 16:45:05.392151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.357724 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7"] Oct 30 16:45:05.439721 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.439695 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.515321 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.472515 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7"] Oct 30 16:45:05.548628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.527042 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4j6q4\"" Oct 30 16:45:05.548926 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.543094 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 16:45:05.549079 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.543942 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 16:45:05.549181 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.543985 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 16:45:05.593616 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.590372 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.593616 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.590509 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.593616 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.590620 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqjvl\" (UniqueName: \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-kube-api-access-tqjvl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.593616 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.590727 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/422f5c36-cc16-4683-ad5c-af1a16c04714-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.593616 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.590864 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.593616 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.590935 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.593616 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.591017 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.649762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.649735 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:45:05.655914 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.655890 6400 scope.go:122] "RemoveContainer" containerID="18b3f060277f07af58281790c92d80c8d61bba4c52d85c4aed5ddc898a7e5007" Oct 30 16:45:05.656315 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:05.656290 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc_odh-ai-gateway-infra(7aecd31a-5a84-4228-8511-4449e12b6ea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" Oct 30 16:45:05.658766 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.658745 6400 scope.go:122] "RemoveContainer" containerID="fc8d5a1a0bd59f64a8207cb14d16a003bc33e96dd2cbb73e827b34f8f8f670bf" Oct 30 16:45:05.659300 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:05.659248 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s_odh-ai-gateway-infra(31d5de4a-8186-4866-800a-a78d45252020)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" podUID="31d5de4a-8186-4866-800a-a78d45252020" Oct 30 16:45:05.659300 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.659280 6400 scope.go:122] "RemoveContainer" containerID="df3348c7a0db1993d0b821e286edb849a98ff5b94645f5686a537d363497207e" Oct 30 16:45:05.659664 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:05.659640 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889645-6czvm_odh-ai-gateway-infra(732a9fa1-7151-44cf-a120-e264fddb5a23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" Oct 30 16:45:05.660798 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.660770 6400 generic.go:356] "Generic (PLEG): container finished" podID="8cb51b6a-7dba-452f-8fea-f90247d603fb" containerID="1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac" exitCode=0 Oct 30 16:45:05.660875 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.660810 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" Oct 30 16:45:05.660911 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.660889 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" event={"ID":"8cb51b6a-7dba-452f-8fea-f90247d603fb","Type":"ContainerDied","Data":"1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac"} Oct 30 16:45:05.661132 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.660926 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9" event={"ID":"8cb51b6a-7dba-452f-8fea-f90247d603fb","Type":"ContainerDied","Data":"5dc95c6b22d5191c587e7cd87f3219fc79ca86fbbaf0f0ac7196b03cbf8e2580"} Oct 30 16:45:05.661132 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.660956 6400 scope.go:122] "RemoveContainer" containerID="1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac" Oct 30 16:45:05.662874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.662849 6400 scope.go:122] "RemoveContainer" containerID="121d0f2d77bdf642e7f57264e46bce04c7a5a84124d562ea4f0108f40083d6dd" Oct 30 16:45:05.663164 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:05.663145 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9_odh-ai-gateway-infra(e900b162-79ab-442e-b0bc-f70844a9165f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" Oct 30 16:45:05.685519 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.684440 6400 scope.go:122] "RemoveContainer" containerID="1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac" Oct 30 16:45:05.685519 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:05.685350 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac\": container with ID starting with 1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac not found: ID does not exist" containerID="1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac" Oct 30 16:45:05.685519 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.685393 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac"} err="failed to get container status \"1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac\": rpc error: code = NotFound desc = could not find container \"1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac\": container with ID starting with 1595bcc7cb23745ec0c378f37463e59fccd6d93c3036dbe16a65f0f336290eac not found: ID does not exist" Oct 30 16:45:05.694580 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.694542 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-metrics-certs\") pod \"8cb51b6a-7dba-452f-8fea-f90247d603fb\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " Oct 30 16:45:05.698904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.694907 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-maas-api-tls\") pod \"8cb51b6a-7dba-452f-8fea-f90247d603fb\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " Oct 30 16:45:05.698904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.694962 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cb51b6a-7dba-452f-8fea-f90247d603fb-kube-api-access-pg69z\" (UniqueName: \"kubernetes.io/projected/8cb51b6a-7dba-452f-8fea-f90247d603fb-kube-api-access-pg69z\") pod \"8cb51b6a-7dba-452f-8fea-f90247d603fb\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " Oct 30 16:45:05.698904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.695074 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cb51b6a-7dba-452f-8fea-f90247d603fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb51b6a-7dba-452f-8fea-f90247d603fb-openshift-service-ca\") pod \"8cb51b6a-7dba-452f-8fea-f90247d603fb\" (UID: \"8cb51b6a-7dba-452f-8fea-f90247d603fb\") " Oct 30 16:45:05.698904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.695381 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.698904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.695472 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqjvl\" (UniqueName: \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-kube-api-access-tqjvl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.698904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.695575 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/422f5c36-cc16-4683-ad5c-af1a16c04714-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.698904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.695729 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.698904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.695785 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.698904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.695854 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.698904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.696041 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.702111 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.699677 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-metrics-certs" pod "8cb51b6a-7dba-452f-8fea-f90247d603fb" (UID: "8cb51b6a-7dba-452f-8fea-f90247d603fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:45:05.702111 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.701958 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.702820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.702792 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.705041 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.705019 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/422f5c36-cc16-4683-ad5c-af1a16c04714-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.756384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.755835 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb51b6a-7dba-452f-8fea-f90247d603fb-kube-api-access-pg69z" pod "8cb51b6a-7dba-452f-8fea-f90247d603fb" (UID: "8cb51b6a-7dba-452f-8fea-f90247d603fb"). InnerVolumeSpecName "kube-api-access-pg69z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:05.756384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.755905 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb51b6a-7dba-452f-8fea-f90247d603fb-openshift-service-ca" pod "8cb51b6a-7dba-452f-8fea-f90247d603fb" (UID: "8cb51b6a-7dba-452f-8fea-f90247d603fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:05.756384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.756029 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-maas-api-tls" pod "8cb51b6a-7dba-452f-8fea-f90247d603fb" (UID: "8cb51b6a-7dba-452f-8fea-f90247d603fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:45:05.756698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.756638 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.756757 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.756723 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.761003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.760978 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqjvl\" (UniqueName: \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-kube-api-access-tqjvl\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.762488 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.762464 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-7fhj7\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:05.799806 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.799733 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:05.799806 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.799809 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pg69z\" (UniqueName: \"kubernetes.io/projected/8cb51b6a-7dba-452f-8fea-f90247d603fb-kube-api-access-pg69z\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:05.800050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.799825 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cb51b6a-7dba-452f-8fea-f90247d603fb-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:05.800050 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.799839 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cb51b6a-7dba-452f-8fea-f90247d603fb-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:05.948012 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:05.947912 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:06.036028 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.009312 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9"] Oct 30 16:45:06.036028 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.032035 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-e4aca49c-689b49dcf7-vtnw9"] Oct 30 16:45:06.295262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.295234 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:45:06.295443 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.295284 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 16:45:06.339353 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.339194 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7"] Oct 30 16:45:06.672546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.672463 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" event={"ID":"422f5c36-cc16-4683-ad5c-af1a16c04714","Type":"ContainerStarted","Data":"9f1517241f415cca08c6c61e5b2951e9829eb6abbe2c7dad59bf1bb081eb29a6"} Oct 30 16:45:06.879717 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.879690 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" Oct 30 16:45:06.946158 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.943346 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31d5de4a-8186-4866-800a-a78d45252020-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d5de4a-8186-4866-800a-a78d45252020-maas-api-service-ca\") pod \"31d5de4a-8186-4866-800a-a78d45252020\" (UID: \"31d5de4a-8186-4866-800a-a78d45252020\") " Oct 30 16:45:06.946158 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.943528 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31d5de4a-8186-4866-800a-a78d45252020-kube-api-access-v4qsb\" (UniqueName: \"kubernetes.io/projected/31d5de4a-8186-4866-800a-a78d45252020-kube-api-access-v4qsb\") pod \"31d5de4a-8186-4866-800a-a78d45252020\" (UID: \"31d5de4a-8186-4866-800a-a78d45252020\") " Oct 30 16:45:06.946158 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.943765 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31d5de4a-8186-4866-800a-a78d45252020-maas-api-service-ca" pod "31d5de4a-8186-4866-800a-a78d45252020" (UID: "31d5de4a-8186-4866-800a-a78d45252020"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:06.946158 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.944286 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31d5de4a-8186-4866-800a-a78d45252020-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:06.946473 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.946263 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31d5de4a-8186-4866-800a-a78d45252020-kube-api-access-v4qsb" pod "31d5de4a-8186-4866-800a-a78d45252020" (UID: "31d5de4a-8186-4866-800a-a78d45252020"). InnerVolumeSpecName "kube-api-access-v4qsb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:06.972453 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:06.972377 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cb51b6a-7dba-452f-8fea-f90247d603fb" path="/var/lib/kubelet/pods/8cb51b6a-7dba-452f-8fea-f90247d603fb/volumes" Oct 30 16:45:07.046669 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:07.046631 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4qsb\" (UniqueName: \"kubernetes.io/projected/31d5de4a-8186-4866-800a-a78d45252020-kube-api-access-v4qsb\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:07.295498 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:07.295462 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" start-of-body= Oct 30 16:45:07.295689 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:07.295524 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.94:15021/healthz/ready\": dial tcp 10.134.0.94:15021: connect: connection refused" Oct 30 16:45:07.687939 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:07.687852 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" event={"ID":"31d5de4a-8186-4866-800a-a78d45252020","Type":"ContainerDied","Data":"dd2ff8fdc2aecb61eac7ba5e738de56acce36b60ada85bd0c21022193c5f1386"} Oct 30 16:45:07.687939 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:07.687927 6400 scope.go:122] "RemoveContainer" containerID="fc8d5a1a0bd59f64a8207cb14d16a003bc33e96dd2cbb73e827b34f8f8f670bf" Oct 30 16:45:07.688374 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:07.688050 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s" Oct 30 16:45:07.751813 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:07.751779 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s"] Oct 30 16:45:07.757534 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:07.757501 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-e4aca49c-29889645-b2d2s"] Oct 30 16:45:08.532765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:08.532735 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:45:08.536662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:08.535495 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:45:08.540831 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:08.538710 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 16:45:08.973025 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:08.972935 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31d5de4a-8186-4866-800a-a78d45252020" path="/var/lib/kubelet/pods/31d5de4a-8186-4866-800a-a78d45252020/volumes" Oct 30 16:45:09.545097 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:09.545061 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:09.545210 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:09.545129 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:09.714126 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:09.714092 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" event={"ID":"422f5c36-cc16-4683-ad5c-af1a16c04714","Type":"ContainerStarted","Data":"a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272"} Oct 30 16:45:09.714328 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:09.714309 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:09.779141 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:09.779084 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" podStartSLOduration=1.573054148 podStartE2EDuration="4.779067124s" totalImagesPullingTime="3.206012976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:45:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:45:09.77588796 +0000 UTC m=+1418.926168949" watchObservedRunningTime="2026-10-30 16:45:09.779067124 +0000 UTC m=+1418.929348114" Oct 30 16:45:10.725186 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:10.725150 6400 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7fhj7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:45:10.725619 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:10.725210 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" podUID="422f5c36-cc16-4683-ad5c-af1a16c04714" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:45:13.114283 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:13.114230 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5"] Oct 30 16:45:13.114829 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:13.114594 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" containerID="cri-o://643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b" gracePeriod=30 Oct 30 16:45:13.724098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:13.724067 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:45:14.960581 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:14.960548 6400 scope.go:122] "RemoveContainer" containerID="121d0f2d77bdf642e7f57264e46bce04c7a5a84124d562ea4f0108f40083d6dd" Oct 30 16:45:15.760969 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:15.760932 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" event={"ID":"e900b162-79ab-442e-b0bc-f70844a9165f","Type":"ContainerStarted","Data":"8ebfd2962e9d0cf43d3766dab01a69c5327b540b85f182609a935d6a010291a8"} Oct 30 16:45:16.804651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:16.804559 6400 generic.go:356] "Generic (PLEG): container finished" podID="e900b162-79ab-442e-b0bc-f70844a9165f" containerID="8ebfd2962e9d0cf43d3766dab01a69c5327b540b85f182609a935d6a010291a8" exitCode=60 Oct 30 16:45:16.805054 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:16.804666 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" event={"ID":"e900b162-79ab-442e-b0bc-f70844a9165f","Type":"ContainerDied","Data":"8ebfd2962e9d0cf43d3766dab01a69c5327b540b85f182609a935d6a010291a8"} Oct 30 16:45:16.805054 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:16.804731 6400 scope.go:122] "RemoveContainer" containerID="121d0f2d77bdf642e7f57264e46bce04c7a5a84124d562ea4f0108f40083d6dd" Oct 30 16:45:16.805441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:16.805420 6400 scope.go:122] "RemoveContainer" containerID="8ebfd2962e9d0cf43d3766dab01a69c5327b540b85f182609a935d6a010291a8" Oct 30 16:45:16.805949 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:16.805918 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9_odh-ai-gateway-infra(e900b162-79ab-442e-b0bc-f70844a9165f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" Oct 30 16:45:16.825671 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:16.816446 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9"] Oct 30 16:45:16.960391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:16.960351 6400 scope.go:122] "RemoveContainer" containerID="df3348c7a0db1993d0b821e286edb849a98ff5b94645f5686a537d363497207e" Oct 30 16:45:16.960584 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:16.960422 6400 scope.go:122] "RemoveContainer" containerID="18b3f060277f07af58281790c92d80c8d61bba4c52d85c4aed5ddc898a7e5007" Oct 30 16:45:17.087703 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:17.087612 6400 patch_prober.go:29] interesting pod/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:45:17.087703 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:17.087668 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:45:17.814791 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:17.814759 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" event={"ID":"732a9fa1-7151-44cf-a120-e264fddb5a23","Type":"ContainerStarted","Data":"84f9529715af21a520794da8bb3c5de92bc3290a1f0ab44972afb1c95809eedc"} Oct 30 16:45:17.818583 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:17.818557 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" event={"ID":"7aecd31a-5a84-4228-8511-4449e12b6ea3","Type":"ContainerStarted","Data":"a07c7025bbab07bf0af32a4a8c1a591c355ec45cf3227df2bc0cafdc669fe568"} Oct 30 16:45:17.987045 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:17.987015 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" Oct 30 16:45:18.103337 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.103248 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e900b162-79ab-442e-b0bc-f70844a9165f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e900b162-79ab-442e-b0bc-f70844a9165f-maas-api-service-ca\") pod \"e900b162-79ab-442e-b0bc-f70844a9165f\" (UID: \"e900b162-79ab-442e-b0bc-f70844a9165f\") " Oct 30 16:45:18.103497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.103360 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e900b162-79ab-442e-b0bc-f70844a9165f-kube-api-access-cxg2k\" (UniqueName: \"kubernetes.io/projected/e900b162-79ab-442e-b0bc-f70844a9165f-kube-api-access-cxg2k\") pod \"e900b162-79ab-442e-b0bc-f70844a9165f\" (UID: \"e900b162-79ab-442e-b0bc-f70844a9165f\") " Oct 30 16:45:18.103714 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.103686 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e900b162-79ab-442e-b0bc-f70844a9165f-maas-api-service-ca" pod "e900b162-79ab-442e-b0bc-f70844a9165f" (UID: "e900b162-79ab-442e-b0bc-f70844a9165f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:18.104456 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.104426 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e900b162-79ab-442e-b0bc-f70844a9165f-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.106173 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.106144 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e900b162-79ab-442e-b0bc-f70844a9165f-kube-api-access-cxg2k" pod "e900b162-79ab-442e-b0bc-f70844a9165f" (UID: "e900b162-79ab-442e-b0bc-f70844a9165f"). InnerVolumeSpecName "kube-api-access-cxg2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:18.205641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.205588 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxg2k\" (UniqueName: \"kubernetes.io/projected/e900b162-79ab-442e-b0bc-f70844a9165f-kube-api-access-cxg2k\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.387486 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.387463 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:45:18.509993 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.509963 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-certs\") pod \"07713c9b-864c-43db-9a1b-f7aef356f607\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " Oct 30 16:45:18.510175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510025 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-data\") pod \"07713c9b-864c-43db-9a1b-f7aef356f607\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " Oct 30 16:45:18.510175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510105 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-credential-socket\") pod \"07713c9b-864c-43db-9a1b-f7aef356f607\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " Oct 30 16:45:18.510175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510152 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-kube-api-access-dpg98\" (UniqueName: \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-kube-api-access-dpg98\") pod \"07713c9b-864c-43db-9a1b-f7aef356f607\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " Oct 30 16:45:18.510344 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510224 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-socket\") pod \"07713c9b-864c-43db-9a1b-f7aef356f607\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " Oct 30 16:45:18.510344 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510279 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07713c9b-864c-43db-9a1b-f7aef356f607-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07713c9b-864c-43db-9a1b-f7aef356f607-istio-podinfo\") pod \"07713c9b-864c-43db-9a1b-f7aef356f607\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " Oct 30 16:45:18.510344 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510280 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-certs" pod "07713c9b-864c-43db-9a1b-f7aef356f607" (UID: "07713c9b-864c-43db-9a1b-f7aef356f607"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:18.510344 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510316 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-data" pod "07713c9b-864c-43db-9a1b-f7aef356f607" (UID: "07713c9b-864c-43db-9a1b-f7aef356f607"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:18.510542 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510347 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-istio-token\" (UniqueName: \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-istio-token\") pod \"07713c9b-864c-43db-9a1b-f7aef356f607\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " Oct 30 16:45:18.510542 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510407 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-envoy\") pod \"07713c9b-864c-43db-9a1b-f7aef356f607\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " Oct 30 16:45:18.510542 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510483 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-credential-socket" pod "07713c9b-864c-43db-9a1b-f7aef356f607" (UID: "07713c9b-864c-43db-9a1b-f7aef356f607"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:18.510730 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510573 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-socket" pod "07713c9b-864c-43db-9a1b-f7aef356f607" (UID: "07713c9b-864c-43db-9a1b-f7aef356f607"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:18.510730 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.510662 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07713c9b-864c-43db-9a1b-f7aef356f607-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07713c9b-864c-43db-9a1b-f7aef356f607-istiod-ca-cert\") pod \"07713c9b-864c-43db-9a1b-f7aef356f607\" (UID: \"07713c9b-864c-43db-9a1b-f7aef356f607\") " Oct 30 16:45:18.511322 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.511294 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07713c9b-864c-43db-9a1b-f7aef356f607-istiod-ca-cert" pod "07713c9b-864c-43db-9a1b-f7aef356f607" (UID: "07713c9b-864c-43db-9a1b-f7aef356f607"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:18.511691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.511670 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07713c9b-864c-43db-9a1b-f7aef356f607-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.511779 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.511696 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.511779 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.511710 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.511779 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.511722 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.511779 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.511734 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.512925 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.512900 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07713c9b-864c-43db-9a1b-f7aef356f607-istio-podinfo" pod "07713c9b-864c-43db-9a1b-f7aef356f607" (UID: "07713c9b-864c-43db-9a1b-f7aef356f607"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:45:18.512925 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.512909 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-istio-token" pod "07713c9b-864c-43db-9a1b-f7aef356f607" (UID: "07713c9b-864c-43db-9a1b-f7aef356f607"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:18.513177 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.513162 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-envoy" pod "07713c9b-864c-43db-9a1b-f7aef356f607" (UID: "07713c9b-864c-43db-9a1b-f7aef356f607"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:45:18.513233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.513174 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-kube-api-access-dpg98" pod "07713c9b-864c-43db-9a1b-f7aef356f607" (UID: "07713c9b-864c-43db-9a1b-f7aef356f607"). InnerVolumeSpecName "kube-api-access-dpg98". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:18.613883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.613851 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpg98\" (UniqueName: \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-kube-api-access-dpg98\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.613883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.613884 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07713c9b-864c-43db-9a1b-f7aef356f607-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.613883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.613893 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07713c9b-864c-43db-9a1b-f7aef356f607-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.614087 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.613902 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07713c9b-864c-43db-9a1b-f7aef356f607-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:18.828114 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.828081 6400 generic.go:356] "Generic (PLEG): container finished" podID="07713c9b-864c-43db-9a1b-f7aef356f607" containerID="643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b" exitCode=0 Oct 30 16:45:18.828563 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.828132 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" Oct 30 16:45:18.828563 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.828168 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" event={"ID":"07713c9b-864c-43db-9a1b-f7aef356f607","Type":"ContainerDied","Data":"643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b"} Oct 30 16:45:18.828563 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.828218 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5" event={"ID":"07713c9b-864c-43db-9a1b-f7aef356f607","Type":"ContainerDied","Data":"2a0f4ab596a21dcb2447cc84769387b82090ebe34d8a50a5f0c6d18084b32b44"} Oct 30 16:45:18.828563 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.828247 6400 scope.go:122] "RemoveContainer" containerID="643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b" Oct 30 16:45:18.830498 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.830477 6400 generic.go:356] "Generic (PLEG): container finished" podID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerID="84f9529715af21a520794da8bb3c5de92bc3290a1f0ab44972afb1c95809eedc" exitCode=60 Oct 30 16:45:18.830596 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.830549 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" event={"ID":"732a9fa1-7151-44cf-a120-e264fddb5a23","Type":"ContainerDied","Data":"84f9529715af21a520794da8bb3c5de92bc3290a1f0ab44972afb1c95809eedc"} Oct 30 16:45:18.831038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.831013 6400 scope.go:122] "RemoveContainer" containerID="84f9529715af21a520794da8bb3c5de92bc3290a1f0ab44972afb1c95809eedc" Oct 30 16:45:18.831366 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:18.831341 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889645-6czvm_odh-ai-gateway-infra(732a9fa1-7151-44cf-a120-e264fddb5a23)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" Oct 30 16:45:18.832271 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.832256 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" Oct 30 16:45:18.832428 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.832278 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9" event={"ID":"e900b162-79ab-442e-b0bc-f70844a9165f","Type":"ContainerDied","Data":"336d0b8b7119ac0559f3009ef3ea15698ffaef6bf37c47d77a9830b2fd9a40f6"} Oct 30 16:45:18.834590 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.834571 6400 generic.go:356] "Generic (PLEG): container finished" podID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerID="a07c7025bbab07bf0af32a4a8c1a591c355ec45cf3227df2bc0cafdc669fe568" exitCode=60 Oct 30 16:45:18.834700 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.834628 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" event={"ID":"7aecd31a-5a84-4228-8511-4449e12b6ea3","Type":"ContainerDied","Data":"a07c7025bbab07bf0af32a4a8c1a591c355ec45cf3227df2bc0cafdc669fe568"} Oct 30 16:45:18.835066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.835048 6400 scope.go:122] "RemoveContainer" containerID="a07c7025bbab07bf0af32a4a8c1a591c355ec45cf3227df2bc0cafdc669fe568" Oct 30 16:45:18.835365 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:18.835345 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc_odh-ai-gateway-infra(7aecd31a-5a84-4228-8511-4449e12b6ea3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" Oct 30 16:45:18.841768 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.841745 6400 scope.go:122] "RemoveContainer" containerID="847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513" Oct 30 16:45:18.852869 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.852852 6400 scope.go:122] "RemoveContainer" containerID="643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b" Oct 30 16:45:18.853220 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:18.853157 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b\": container with ID starting with 643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b not found: ID does not exist" containerID="643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b" Oct 30 16:45:18.853282 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.853232 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b"} err="failed to get container status \"643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b\": rpc error: code = NotFound desc = could not find container \"643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b\": container with ID starting with 643aa2a23a362ea28272d0cf2cf6b4b4b14e0af1a282e7648a0c3475d773d05b not found: ID does not exist" Oct 30 16:45:18.853282 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.853249 6400 scope.go:122] "RemoveContainer" containerID="847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513" Oct 30 16:45:18.853496 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:18.853477 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513\": container with ID starting with 847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513 not found: ID does not exist" containerID="847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513" Oct 30 16:45:18.853545 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.853504 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513"} err="failed to get container status \"847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513\": rpc error: code = NotFound desc = could not find container \"847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513\": container with ID starting with 847544a33e0afdf7a7ba50c043d06f3989c8c89d4916622cc23f0ac0311bf513 not found: ID does not exist" Oct 30 16:45:18.853545 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.853520 6400 scope.go:122] "RemoveContainer" containerID="df3348c7a0db1993d0b821e286edb849a98ff5b94645f5686a537d363497207e" Oct 30 16:45:18.866411 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.866387 6400 scope.go:122] "RemoveContainer" containerID="8ebfd2962e9d0cf43d3766dab01a69c5327b540b85f182609a935d6a010291a8" Oct 30 16:45:18.866561 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.866542 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm"] Oct 30 16:45:18.878763 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.878740 6400 scope.go:122] "RemoveContainer" containerID="18b3f060277f07af58281790c92d80c8d61bba4c52d85c4aed5ddc898a7e5007" Oct 30 16:45:18.899677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.899650 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc"] Oct 30 16:45:18.971955 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.971923 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5"] Oct 30 16:45:18.978494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:18.978463 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-e4aca49c-openshift-default-5b56b4f88f-bfjr5"] Oct 30 16:45:19.022817 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:19.022781 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9"] Oct 30 16:45:19.035016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:19.034986 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-263f00-29889645-99xl9"] Oct 30 16:45:20.140747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.140718 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" Oct 30 16:45:20.172427 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.172395 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" Oct 30 16:45:20.246084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.246047 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7aecd31a-5a84-4228-8511-4449e12b6ea3-kube-api-access-rsk9b\" (UniqueName: \"kubernetes.io/projected/7aecd31a-5a84-4228-8511-4449e12b6ea3-kube-api-access-rsk9b\") pod \"7aecd31a-5a84-4228-8511-4449e12b6ea3\" (UID: \"7aecd31a-5a84-4228-8511-4449e12b6ea3\") " Oct 30 16:45:20.246084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.246089 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7aecd31a-5a84-4228-8511-4449e12b6ea3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aecd31a-5a84-4228-8511-4449e12b6ea3-maas-api-service-ca\") pod \"7aecd31a-5a84-4228-8511-4449e12b6ea3\" (UID: \"7aecd31a-5a84-4228-8511-4449e12b6ea3\") " Oct 30 16:45:20.246352 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.246327 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/732a9fa1-7151-44cf-a120-e264fddb5a23-kube-api-access-dwd6m\" (UniqueName: \"kubernetes.io/projected/732a9fa1-7151-44cf-a120-e264fddb5a23-kube-api-access-dwd6m\") pod \"732a9fa1-7151-44cf-a120-e264fddb5a23\" (UID: \"732a9fa1-7151-44cf-a120-e264fddb5a23\") " Oct 30 16:45:20.246416 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.246366 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/732a9fa1-7151-44cf-a120-e264fddb5a23-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732a9fa1-7151-44cf-a120-e264fddb5a23-maas-api-service-ca\") pod \"732a9fa1-7151-44cf-a120-e264fddb5a23\" (UID: \"732a9fa1-7151-44cf-a120-e264fddb5a23\") " Oct 30 16:45:20.246563 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.246535 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aecd31a-5a84-4228-8511-4449e12b6ea3-maas-api-service-ca" pod "7aecd31a-5a84-4228-8511-4449e12b6ea3" (UID: "7aecd31a-5a84-4228-8511-4449e12b6ea3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:20.246949 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.246860 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/732a9fa1-7151-44cf-a120-e264fddb5a23-maas-api-service-ca" pod "732a9fa1-7151-44cf-a120-e264fddb5a23" (UID: "732a9fa1-7151-44cf-a120-e264fddb5a23"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:45:20.247252 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.247233 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/732a9fa1-7151-44cf-a120-e264fddb5a23-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:20.247317 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.247262 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aecd31a-5a84-4228-8511-4449e12b6ea3-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:20.248811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.248776 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7aecd31a-5a84-4228-8511-4449e12b6ea3-kube-api-access-rsk9b" pod "7aecd31a-5a84-4228-8511-4449e12b6ea3" (UID: "7aecd31a-5a84-4228-8511-4449e12b6ea3"). InnerVolumeSpecName "kube-api-access-rsk9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:20.249118 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.249092 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/732a9fa1-7151-44cf-a120-e264fddb5a23-kube-api-access-dwd6m" pod "732a9fa1-7151-44cf-a120-e264fddb5a23" (UID: "732a9fa1-7151-44cf-a120-e264fddb5a23"). InnerVolumeSpecName "kube-api-access-dwd6m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:45:20.349410 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.349374 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dwd6m\" (UniqueName: \"kubernetes.io/projected/732a9fa1-7151-44cf-a120-e264fddb5a23-kube-api-access-dwd6m\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:20.349410 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.349404 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsk9b\" (UniqueName: \"kubernetes.io/projected/7aecd31a-5a84-4228-8511-4449e12b6ea3-kube-api-access-rsk9b\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:45:20.863991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.863957 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" Oct 30 16:45:20.864223 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.864187 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc" event={"ID":"7aecd31a-5a84-4228-8511-4449e12b6ea3","Type":"ContainerDied","Data":"bbcaf0795f7f9b570f460608720b2a1d577693c51808cf66b2eea47bb17abbc3"} Oct 30 16:45:20.864381 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.864296 6400 scope.go:122] "RemoveContainer" containerID="a07c7025bbab07bf0af32a4a8c1a591c355ec45cf3227df2bc0cafdc669fe568" Oct 30 16:45:20.865982 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.865891 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" event={"ID":"732a9fa1-7151-44cf-a120-e264fddb5a23","Type":"ContainerDied","Data":"779e39c4d98e730b13adb431058e141b6890e12457313c4b0b1e3bf15c8994b9"} Oct 30 16:45:20.866066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.865981 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm" Oct 30 16:45:20.877940 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.877915 6400 scope.go:122] "RemoveContainer" containerID="84f9529715af21a520794da8bb3c5de92bc3290a1f0ab44972afb1c95809eedc" Oct 30 16:45:20.932384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.932356 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc"] Oct 30 16:45:20.936844 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.936813 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-a8c22d-29889645-w8psc"] Oct 30 16:45:20.976654 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.976618 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" path="/var/lib/kubelet/pods/07713c9b-864c-43db-9a1b-f7aef356f607/volumes" Oct 30 16:45:20.977395 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.977374 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" path="/var/lib/kubelet/pods/7aecd31a-5a84-4228-8511-4449e12b6ea3/volumes" Oct 30 16:45:20.977872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.977856 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" path="/var/lib/kubelet/pods/e900b162-79ab-442e-b0bc-f70844a9165f/volumes" Oct 30 16:45:20.978499 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.978484 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm"] Oct 30 16:45:20.978557 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:20.978516 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889645-6czvm"] Oct 30 16:45:23.015626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:23.015201 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" path="/var/lib/kubelet/pods/732a9fa1-7151-44cf-a120-e264fddb5a23/volumes" Oct 30 16:45:24.611800 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:24.611763 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:45:43.432329 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:43.432104 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:43.451068 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:43.451023 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:43.528046 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:43.528000 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:43.578931 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:43.574784 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:43.578931 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:43.576656 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:43.578931 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:43.578239 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:43.659623 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:43.632765 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:43.860955 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:43.836703 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:43.860955 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:43.837178 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 16:45:44.160198 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:44.160100 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:53.166857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:53.166759 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:45:53.166857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:53.166830 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:54.158323 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.158237 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:45:54.158323 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.158296 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:45:54.376417 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.376375 6400 generic.go:356] "Generic (PLEG): container finished" podID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerID="300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb" exitCode=0 Oct 30 16:45:54.376934 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.376459 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" event={"ID":"a0a964e5-301d-4f97-b692-f2c9797276c4","Type":"ContainerDied","Data":"300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb"} Oct 30 16:45:54.377596 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.377571 6400 scope.go:122] "RemoveContainer" containerID="300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb" Oct 30 16:45:54.382304 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.382254 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.382415 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.382363 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.382480 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.382421 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.383255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.382750 6400 generic.go:356] "Generic (PLEG): container finished" podID="e7623e9c-0154-4059-bf52-027533652d9e" containerID="40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47" exitCode=0 Oct 30 16:45:54.383255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.382931 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" event={"ID":"e7623e9c-0154-4059-bf52-027533652d9e","Type":"ContainerDied","Data":"40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47"} Oct 30 16:45:54.383503 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.383476 6400 scope.go:122] "RemoveContainer" containerID="40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47" Oct 30 16:45:54.386580 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.386538 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.386723 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.386653 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.386723 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.386707 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.390142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.389482 6400 generic.go:356] "Generic (PLEG): container finished" podID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerID="9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b" exitCode=0 Oct 30 16:45:54.390142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.389696 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" event={"ID":"6fcac11b-bb6f-4004-8830-e52f952f08a5","Type":"ContainerDied","Data":"9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b"} Oct 30 16:45:54.390142 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.390102 6400 scope.go:122] "RemoveContainer" containerID="9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b" Oct 30 16:45:54.403483 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.403387 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.403624 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.403484 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.403624 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:54.403533 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:45:54.727725 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:45:54.723670 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:45:55.461627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:55.458087 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" event={"ID":"e7623e9c-0154-4059-bf52-027533652d9e","Type":"ContainerStarted","Data":"ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93"} Oct 30 16:45:55.473633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:55.472941 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" event={"ID":"6fcac11b-bb6f-4004-8830-e52f952f08a5","Type":"ContainerStarted","Data":"f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df"} Oct 30 16:45:55.508117 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:55.507294 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" event={"ID":"a0a964e5-301d-4f97-b692-f2c9797276c4","Type":"ContainerStarted","Data":"c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946"} Oct 30 16:45:55.545973 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:55.538049 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:45:55.545973 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:55.538177 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:45:55.545973 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:55.538239 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:45:56.098581 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:56.098554 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:45:56.098887 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:56.098870 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 16:45:56.099013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:56.098993 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 16:45:56.324027 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:56.323994 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:45:56.324379 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:56.324355 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:45:56.324594 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:56.324566 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:45:56.500747 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:56.500716 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:45:56.540627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:56.540049 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:45:56.540627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:56.540151 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:45:57.108739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:57.103009 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 16:45:57.108739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:57.103070 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 16:45:57.328629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:57.328421 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:45:57.328629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:57.328500 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:45:57.536054 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:57.536012 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:45:57.540920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:57.540884 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:45:58.132739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.132031 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 16:45:58.132739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.132093 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 16:45:58.321630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.320988 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:45:58.321630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.321099 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:45:58.621632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.615678 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:58.621632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.615981 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:58.621632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.616620 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:45:58.720979 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.667013 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:58.720979 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.687592 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:45:58.720979 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.687672 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:45:58.878756 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:58.878254 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:59.106788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.106751 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 16:45:59.106985 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.106811 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 16:45:59.324160 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.324106 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:45:59.330757 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.330697 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:45:59.505626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.480318 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": context deadline exceeded" Oct 30 16:45:59.505626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.503222 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:59.595082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.594825 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:45:59.595082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.594897 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:45:59.618854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.618805 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:59.638669 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.638638 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:45:59.652626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.646411 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:59.697334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.697289 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": context deadline exceeded" Oct 30 16:45:59.922811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.922694 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:59.958151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.958099 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:45:59.981722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:45:59.981490 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:46:00.182300 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.181268 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 16:46:00.182300 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.181329 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 16:46:00.399733 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.399675 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:46:00.400134 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.400102 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:46:00.603251 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.603191 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" probeResult="failure" output="Get \"https://10.134.0.54:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:00.604851 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.604769 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:46:00.654624 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.650923 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 16:46:00.661839 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.661788 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:46:00.662104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.662076 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:46:00.662241 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.661961 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:00.684187 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.666100 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:00.732630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:00.731296 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 16:46:01.207803 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:01.207760 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 30 16:46:01.207997 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:01.207841 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 30 16:46:01.308645 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:01.308595 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 16:46:01.308918 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:01.308902 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 16:46:01.343749 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:01.332396 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body= Oct 30 16:46:01.379371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:01.379273 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" Oct 30 16:46:01.535899 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:01.535864 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" start-of-body= Oct 30 16:46:01.536117 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:01.535925 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.98:15021/healthz/ready\": dial tcp 10.134.0.98:15021: connect: connection refused" Oct 30 16:46:02.105466 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:02.105431 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:46:02.105984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:02.105808 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:46:02.106914 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:02.106872 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:46:02.325800 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:02.325773 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:46:02.326922 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:02.326901 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:46:02.540745 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:02.540711 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:46:02.541038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:02.541004 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:46:02.541829 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:02.541813 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 16:46:06.940138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.940106 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b"] Oct 30 16:46:06.940986 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.940959 6400 cpu_manager.go:414] "containerMap: removing container" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerName="cleanup" Oct 30 16:46:06.940986 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.940985 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941003 6400 cpu_manager.go:414] "containerMap: removing container" podUID="8cb51b6a-7dba-452f-8fea-f90247d603fb" containerName="maas-api" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941010 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cb51b6a-7dba-452f-8fea-f90247d603fb" containerName="maas-api" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941059 6400 cpu_manager.go:414] "containerMap: removing container" podUID="31d5de4a-8186-4866-800a-a78d45252020" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941068 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d5de4a-8186-4866-800a-a78d45252020" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941084 6400 cpu_manager.go:414] "containerMap: removing container" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941093 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941104 6400 cpu_manager.go:414] "containerMap: removing container" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941111 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941121 6400 cpu_manager.go:414] "containerMap: removing container" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941127 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941152 6400 cpu_manager.go:414] "containerMap: removing container" podUID="31d5de4a-8186-4866-800a-a78d45252020" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941159 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31d5de4a-8186-4866-800a-a78d45252020" containerName="cleanup" Oct 30 16:46:06.941164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941168 6400 cpu_manager.go:414] "containerMap: removing container" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941175 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941191 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941195 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941207 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941211 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941218 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941223 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941237 6400 cpu_manager.go:414] "containerMap: removing container" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941241 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941347 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941356 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941363 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e900b162-79ab-442e-b0bc-f70844a9165f" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941372 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941376 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941383 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cb51b6a-7dba-452f-8fea-f90247d603fb" containerName="maas-api" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941387 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941396 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d5de4a-8186-4866-800a-a78d45252020" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941402 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941408 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941413 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerName="cleanup" Oct 30 16:46:06.941564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.941420 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="31d5de4a-8186-4866-800a-a78d45252020" containerName="cleanup" Oct 30 16:46:06.944706 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.944683 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:06.965455 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.965431 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-968df4-serving-cert\"" Oct 30 16:46:06.972962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:06.972926 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b"] Oct 30 16:46:07.024037 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.024002 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-metrics-certs\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.024206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.024048 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-maas-api-tls\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.024206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.024178 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkzjg\" (UniqueName: \"kubernetes.io/projected/715f3a38-e8db-443f-9490-3ad2b99370ea-kube-api-access-bkzjg\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.024356 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.024340 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715f3a38-e8db-443f-9490-3ad2b99370ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.126293 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.126249 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-maas-api-tls\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.126515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.126371 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkzjg\" (UniqueName: \"kubernetes.io/projected/715f3a38-e8db-443f-9490-3ad2b99370ea-kube-api-access-bkzjg\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.126644 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.126557 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715f3a38-e8db-443f-9490-3ad2b99370ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.127066 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:46:07.127040 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-serving-cert: secret "maas-api-e2e-api-a-968df4-serving-cert" not found Oct 30 16:46:07.127808 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:46:07.127789 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-maas-api-tls podName:715f3a38-e8db-443f-9490-3ad2b99370ea nodeName:}" failed. No retries permitted until 2026-10-30 16:46:07.627766541 +0000 UTC m=+1476.778047506 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-maas-api-tls") pod "maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" (UID: "715f3a38-e8db-443f-9490-3ad2b99370ea") : secret "maas-api-e2e-api-a-968df4-serving-cert" not found Oct 30 16:46:07.128103 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.128069 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-metrics-certs\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.128206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.128073 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715f3a38-e8db-443f-9490-3ad2b99370ea-openshift-service-ca\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.132775 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.132725 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-metrics-certs\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.152188 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.152154 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkzjg\" (UniqueName: \"kubernetes.io/projected/715f3a38-e8db-443f-9490-3ad2b99370ea-kube-api-access-bkzjg\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.635701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.635671 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-maas-api-tls\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.638497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.638468 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-maas-api-tls\") pod \"maas-api-e2e-api-a-968df4-6f49dbf945-nd49b\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:07.862182 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:07.861434 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:08.219780 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:46:08.219736 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod715f3a38_e8db_443f_9490_3ad2b99370ea.slice/crio-89b2af90648425e88461b2d6338851d0ddf87b0c9f134079de3dcd0a72189dfa WatchSource:0}: Error finding container 89b2af90648425e88461b2d6338851d0ddf87b0c9f134079de3dcd0a72189dfa: Status 404 returned error can't find the container with id 89b2af90648425e88461b2d6338851d0ddf87b0c9f134079de3dcd0a72189dfa Oct 30 16:46:08.231754 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:08.230632 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b"] Oct 30 16:46:08.396013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:08.395135 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" event={"ID":"715f3a38-e8db-443f-9490-3ad2b99370ea","Type":"ContainerStarted","Data":"89b2af90648425e88461b2d6338851d0ddf87b0c9f134079de3dcd0a72189dfa"} Oct 30 16:46:09.404164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:09.404131 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" event={"ID":"715f3a38-e8db-443f-9490-3ad2b99370ea","Type":"ContainerStarted","Data":"94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8"} Oct 30 16:46:09.404676 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:09.404328 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:09.443738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:09.443672 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" podStartSLOduration=3.064400091 podStartE2EDuration="3.443656867s" totalImagesPullingTime="379.256776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:09.441670155 +0000 UTC m=+1478.591951136" watchObservedRunningTime="2026-10-30 16:46:09.443656867 +0000 UTC m=+1478.593937856" Oct 30 16:46:15.483629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:15.471869 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:46:17.065628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:17.060492 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:17.511994 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:17.511939 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" probeResult="failure" output="Get \"https://10.134.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:17.612764 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:17.612710 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:17.613443 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:17.613410 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 30 16:46:17.626718 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:17.626678 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": context deadline exceeded" Oct 30 16:46:17.688950 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:17.688218 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" probeResult="failure" output="Get \"https://10.134.0.56:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:17.818731 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:17.817709 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:17.886848 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:17.886785 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": context deadline exceeded" Oct 30 16:46:17.901761 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:17.901711 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:46:18.074806 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:18.074701 6400 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/health\": context deadline exceeded" Oct 30 16:46:24.604105 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:46:24.604063 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:46:25.687762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:25.687661 6400 generic.go:356] "Generic (PLEG): container finished" podID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerID="2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393" exitCode=0 Oct 30 16:46:25.687762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:25.687721 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" event={"ID":"fc4a7872-45cb-498b-a8a4-92eeef9c76b7","Type":"ContainerDied","Data":"2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393"} Oct 30 16:46:25.688224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:25.688175 6400 scope.go:122] "RemoveContainer" containerID="2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393" Oct 30 16:46:25.690069 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:25.690038 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:25.690145 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:25.690107 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:25.690145 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:25.690138 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:26.229795 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.229762 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w"] Oct 30 16:46:26.230422 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.230407 6400 cpu_manager.go:414] "containerMap: removing container" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerName="cleanup" Oct 30 16:46:26.230472 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.230424 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="732a9fa1-7151-44cf-a120-e264fddb5a23" containerName="cleanup" Oct 30 16:46:26.230472 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.230436 6400 cpu_manager.go:414] "containerMap: removing container" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerName="cleanup" Oct 30 16:46:26.230472 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.230443 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerName="cleanup" Oct 30 16:46:26.230569 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.230560 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7aecd31a-5a84-4228-8511-4449e12b6ea3" containerName="cleanup" Oct 30 16:46:26.230569 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.230568 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="07713c9b-864c-43db-9a1b-f7aef356f607" containerName="istio-proxy" Oct 30 16:46:26.233317 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.233298 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.258650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.255840 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w"] Oct 30 16:46:26.278584 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.277679 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-c8db42-serving-cert\"" Oct 30 16:46:26.317389 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.317361 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6m862\" (UniqueName: \"kubernetes.io/projected/89164063-6e0d-4dc3-9e26-30f9c0557c05-kube-api-access-6m862\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.317555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.317476 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.317713 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.317696 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.317838 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.317818 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89164063-6e0d-4dc3-9e26-30f9c0557c05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.419922 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.419886 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6m862\" (UniqueName: \"kubernetes.io/projected/89164063-6e0d-4dc3-9e26-30f9c0557c05-kube-api-access-6m862\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.420117 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.419969 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.420221 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.420201 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.420852 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.420789 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89164063-6e0d-4dc3-9e26-30f9c0557c05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.421762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.421735 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89164063-6e0d-4dc3-9e26-30f9c0557c05-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.423052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.423024 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.423166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.423036 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.441471 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.441441 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6m862\" (UniqueName: \"kubernetes.io/projected/89164063-6e0d-4dc3-9e26-30f9c0557c05-kube-api-access-6m862\") pod \"maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.548882 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.548841 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:26.703660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.703624 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" event={"ID":"fc4a7872-45cb-498b-a8a4-92eeef9c76b7","Type":"ContainerStarted","Data":"b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506"} Oct 30 16:46:26.886889 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:26.886864 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w"] Oct 30 16:46:26.888288 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:46:26.888260 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89164063_6e0d_4dc3_9e26_30f9c0557c05.slice/crio-db8492758a4729d6134d44a0ac6d04e9544dc3e00c5d6f0f049555260ef604b9 WatchSource:0}: Error finding container db8492758a4729d6134d44a0ac6d04e9544dc3e00c5d6f0f049555260ef604b9: Status 404 returned error can't find the container with id db8492758a4729d6134d44a0ac6d04e9544dc3e00c5d6f0f049555260ef604b9 Oct 30 16:46:27.038147 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:27.037869 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:46:27.300026 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:27.299987 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:46:27.301221 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:27.301194 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:46:27.301357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:27.301263 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 16:46:27.741190 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:27.741148 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" event={"ID":"89164063-6e0d-4dc3-9e26-30f9c0557c05","Type":"ContainerStarted","Data":"99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca"} Oct 30 16:46:27.741751 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:27.741205 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" event={"ID":"89164063-6e0d-4dc3-9e26-30f9c0557c05","Type":"ContainerStarted","Data":"db8492758a4729d6134d44a0ac6d04e9544dc3e00c5d6f0f049555260ef604b9"} Oct 30 16:46:27.741751 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:27.741249 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:27.781152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:27.781100 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" podStartSLOduration=1.342978031 podStartE2EDuration="1.781085958s" totalImagesPullingTime="438.107927ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:27.778172445 +0000 UTC m=+1496.928453457" watchObservedRunningTime="2026-10-30 16:46:27.781085958 +0000 UTC m=+1496.931366945" Oct 30 16:46:28.300038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:28.300005 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:46:28.300314 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:28.300057 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 16:46:29.299927 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:29.299888 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:46:29.300355 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:29.299948 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 16:46:30.300391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:30.300348 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:46:30.300962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:30.300415 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 16:46:31.299635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.299583 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:46:31.299827 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.299663 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 16:46:31.364576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.364544 6400 scope.go:122] "RemoveContainer" containerID="a0a9233e0e95ea67f206aa606788b2ec941dfa7231af6a575a21933333a1e633" Oct 30 16:46:31.439994 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.439956 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x"] Oct 30 16:46:31.444910 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.444886 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.464092 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.461818 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-661509-serving-cert\"" Oct 30 16:46:31.479627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.478835 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x"] Oct 30 16:46:31.556162 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.556071 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ef9893-a9f8-47b8-82cc-872188fbc467-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.556314 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.556217 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.556390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.556369 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.556672 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.556656 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvzrd\" (UniqueName: \"kubernetes.io/projected/80ef9893-a9f8-47b8-82cc-872188fbc467-kube-api-access-tvzrd\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.663532 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.663488 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.665669 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.665640 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.678886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.677936 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvzrd\" (UniqueName: \"kubernetes.io/projected/80ef9893-a9f8-47b8-82cc-872188fbc467-kube-api-access-tvzrd\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.678886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.678157 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ef9893-a9f8-47b8-82cc-872188fbc467-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.680670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.679367 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ef9893-a9f8-47b8-82cc-872188fbc467-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.695627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.690384 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.695627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.691186 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.708668 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.708627 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvzrd\" (UniqueName: \"kubernetes.io/projected/80ef9893-a9f8-47b8-82cc-872188fbc467-kube-api-access-tvzrd\") pod \"maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:31.767923 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:31.766496 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:32.143821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:32.143787 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x"] Oct 30 16:46:32.150088 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:46:32.147341 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80ef9893_a9f8_47b8_82cc_872188fbc467.slice/crio-c42346a0626c31d6842ce17cc87c13d90e019ea1c5dc0337d57c7e989c4950d4 WatchSource:0}: Error finding container c42346a0626c31d6842ce17cc87c13d90e019ea1c5dc0337d57c7e989c4950d4: Status 404 returned error can't find the container with id c42346a0626c31d6842ce17cc87c13d90e019ea1c5dc0337d57c7e989c4950d4 Oct 30 16:46:32.300250 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:32.300215 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 30 16:46:32.300435 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:32.300290 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 30 16:46:32.828501 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:32.828442 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" event={"ID":"80ef9893-a9f8-47b8-82cc-872188fbc467","Type":"ContainerStarted","Data":"21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078"} Oct 30 16:46:32.829067 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:32.828521 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" event={"ID":"80ef9893-a9f8-47b8-82cc-872188fbc467","Type":"ContainerStarted","Data":"c42346a0626c31d6842ce17cc87c13d90e019ea1c5dc0337d57c7e989c4950d4"} Oct 30 16:46:32.829067 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:32.828576 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:32.866934 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:32.866878 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" podStartSLOduration=1.561191006 podStartE2EDuration="1.866858742s" totalImagesPullingTime="305.667736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:32.866227152 +0000 UTC m=+1502.016508137" watchObservedRunningTime="2026-10-30 16:46:32.866858742 +0000 UTC m=+1502.017139732" Oct 30 16:46:33.304784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:33.304755 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:46:33.305891 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:33.305870 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 16:46:33.777836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:33.777807 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 16:46:34.852744 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.852711 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b"] Oct 30 16:46:34.857088 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.857062 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:34.871842 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.871815 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:34.872016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.871858 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:34.872016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.871944 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a331a4b1-0c84-4850-a289-beda7882adbf-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:34.872016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.871982 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:34.872016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.872005 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:34.872238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.872184 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:34.872308 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.872240 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a331a4b1-0c84-4850-a289-beda7882adbf-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:34.872365 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.872334 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:34.872420 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.872393 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pp79\" (UniqueName: \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-kube-api-access-6pp79\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:34.884556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:34.884525 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b"] Oct 30 16:46:35.015330 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.015290 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.022517 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.021156 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.022727 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.022489 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pp79\" (UniqueName: \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-kube-api-access-6pp79\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.022798 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.022744 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.022852 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.022806 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.023036 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.022858 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a331a4b1-0c84-4850-a289-beda7882adbf-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.023036 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.022917 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.023036 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.022969 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.023195 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.023158 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.024967 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.024122 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a331a4b1-0c84-4850-a289-beda7882adbf-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.024967 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.024376 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.024967 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.024616 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.024967 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.024879 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a331a4b1-0c84-4850-a289-beda7882adbf-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.026033 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.026012 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.075707 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.075136 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.092886 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.086892 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pp79\" (UniqueName: \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-kube-api-access-6pp79\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.096515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.096448 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.105756 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.098997 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a331a4b1-0c84-4850-a289-beda7882adbf-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.174071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.174042 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:35.540010 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.539974 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:35.540149 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.540060 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:35.540149 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.540105 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:35.547388 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.541829 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b"] Oct 30 16:46:35.871218 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.871060 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" event={"ID":"a331a4b1-0c84-4850-a289-beda7882adbf","Type":"ContainerStarted","Data":"51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c"} Oct 30 16:46:35.871218 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.871140 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" event={"ID":"a331a4b1-0c84-4850-a289-beda7882adbf","Type":"ContainerStarted","Data":"2ae2c3b61e352a5b2d48acb3706bae550a5de3f352bf040c36bd6001a7a66167"} Oct 30 16:46:35.916927 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:35.916875 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" podStartSLOduration=1.916859687 podStartE2EDuration="1.916859687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:35.914898544 +0000 UTC m=+1505.065179533" watchObservedRunningTime="2026-10-30 16:46:35.916859687 +0000 UTC m=+1505.067140772" Oct 30 16:46:36.180858 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:36.175993 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:36.181213 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:36.181182 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:46:36.181368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:36.181344 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 16:46:37.175016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:37.174981 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:46:37.175405 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:37.175039 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 16:46:38.176182 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:38.176134 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:46:38.177112 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:38.177077 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 16:46:38.869895 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:38.869866 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 16:46:39.175238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:39.175146 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:46:39.175238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:39.175209 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 16:46:40.175708 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:40.175674 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:46:40.176133 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:40.175725 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 16:46:41.175467 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:41.175434 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" start-of-body= Oct 30 16:46:41.175715 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:41.175488 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.109:15021/healthz/ready\": dial tcp 10.134.0.109:15021: connect: connection refused" Oct 30 16:46:42.179940 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:42.179912 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:42.180351 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:42.180331 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:42.180980 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:42.180963 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:46:45.957086 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:45.957043 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr"] Oct 30 16:46:45.963204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:45.963177 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:45.982298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:45.982272 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr"] Oct 30 16:46:45.988487 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:45.988464 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b8af660d-openshift-default-dockercfg-vhvv6\"" Oct 30 16:46:46.038565 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.038521 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.038917 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.038890 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.039099 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.039079 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twgvp\" (UniqueName: \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-kube-api-access-twgvp\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.039350 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.039317 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.039714 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.039679 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1899f96-b860-4c71-a10b-c36ee2587ace-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.039849 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.039833 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.040127 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.040103 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.040258 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.040242 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.040376 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.040361 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142085 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142050 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142337 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142119 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142337 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142167 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twgvp\" (UniqueName: \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-kube-api-access-twgvp\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142337 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142238 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142337 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142318 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1899f96-b860-4c71-a10b-c36ee2587ace-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142440 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142443 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142485 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142687 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142739 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142726 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.142847 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.142777 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.143077 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.143049 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1899f96-b860-4c71-a10b-c36ee2587ace-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.143362 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.143153 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.143362 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.143278 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.145070 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.145050 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.145199 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.145102 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.165547 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.165515 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.165719 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.165573 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twgvp\" (UniqueName: \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-kube-api-access-twgvp\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.278913 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.278879 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:46.608831 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:46:46.608799 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1899f96_b860_4c71_a10b_c36ee2587ace.slice/crio-07bbe4a297a9de4efe21c9dcd1f4741b45a9650dad3e2c046bfdb84b6c7b9e60 WatchSource:0}: Error finding container 07bbe4a297a9de4efe21c9dcd1f4741b45a9650dad3e2c046bfdb84b6c7b9e60: Status 404 returned error can't find the container with id 07bbe4a297a9de4efe21c9dcd1f4741b45a9650dad3e2c046bfdb84b6c7b9e60 Oct 30 16:46:46.610062 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.610039 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr"] Oct 30 16:46:46.611791 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.611760 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:46.611858 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.611826 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:46.611905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.611856 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:46:46.995697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.995658 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" event={"ID":"f1899f96-b860-4c71-a10b-c36ee2587ace","Type":"ContainerStarted","Data":"04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759"} Oct 30 16:46:46.996185 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:46.995719 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" event={"ID":"f1899f96-b860-4c71-a10b-c36ee2587ace","Type":"ContainerStarted","Data":"07bbe4a297a9de4efe21c9dcd1f4741b45a9650dad3e2c046bfdb84b6c7b9e60"} Oct 30 16:46:47.041424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:47.041364 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" podStartSLOduration=2.041344508 podStartE2EDuration="2.041344508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:47.039411529 +0000 UTC m=+1516.189692519" watchObservedRunningTime="2026-10-30 16:46:47.041344508 +0000 UTC m=+1516.191625498" Oct 30 16:46:47.279529 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:47.279436 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:47.281450 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:47.281420 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:46:47.281585 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:47.281482 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 16:46:48.279744 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:48.279700 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:46:48.280233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:48.279770 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 16:46:49.280033 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:49.279999 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:46:49.280421 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:49.280054 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 16:46:50.279978 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:50.279939 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:46:50.280174 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:50.280006 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 16:46:51.151811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.151779 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4"] Oct 30 16:46:51.155568 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.155549 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.170717 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.170687 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-b315d4-serving-cert\"" Oct 30 16:46:51.187372 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.187345 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4"] Oct 30 16:46:51.191179 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.191151 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-maas-api-tls\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.191577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.191556 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnldf\" (UniqueName: \"kubernetes.io/projected/d0d8af8b-c70a-4a25-bd13-577b099058ec-kube-api-access-gnldf\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.191855 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.191834 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d8af8b-c70a-4a25-bd13-577b099058ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.192197 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.192180 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-metrics-certs\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.279784 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.279753 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:46:51.280022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.279995 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 16:46:51.293903 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.293875 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnldf\" (UniqueName: \"kubernetes.io/projected/d0d8af8b-c70a-4a25-bd13-577b099058ec-kube-api-access-gnldf\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.294540 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.294516 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d8af8b-c70a-4a25-bd13-577b099058ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.294853 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.294832 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-metrics-certs\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.295022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.295006 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-maas-api-tls\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.295165 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.295140 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d8af8b-c70a-4a25-bd13-577b099058ec-openshift-service-ca\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.297531 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.297512 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-maas-api-tls\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.301564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.301542 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-metrics-certs\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.343209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.343153 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnldf\" (UniqueName: \"kubernetes.io/projected/d0d8af8b-c70a-4a25-bd13-577b099058ec-kube-api-access-gnldf\") pod \"maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.471578 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.471492 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:51.848470 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:46:51.848441 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0d8af8b_c70a_4a25_bd13_577b099058ec.slice/crio-cebaac585d3463e32f341882a36c38c727a9c07ffe27764c50615493e3241b61 WatchSource:0}: Error finding container cebaac585d3463e32f341882a36c38c727a9c07ffe27764c50615493e3241b61: Status 404 returned error can't find the container with id cebaac585d3463e32f341882a36c38c727a9c07ffe27764c50615493e3241b61 Oct 30 16:46:51.850801 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:51.850775 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4"] Oct 30 16:46:52.092902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:52.092854 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" event={"ID":"d0d8af8b-c70a-4a25-bd13-577b099058ec","Type":"ContainerStarted","Data":"cebaac585d3463e32f341882a36c38c727a9c07ffe27764c50615493e3241b61"} Oct 30 16:46:52.279345 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:52.279316 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:46:52.279532 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:52.279368 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 16:46:53.103368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:53.103339 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" event={"ID":"d0d8af8b-c70a-4a25-bd13-577b099058ec","Type":"ContainerStarted","Data":"86ec40e3392e9aa7e58971c203432cb86b765c1f3e57f206cca9fc8aaacc734b"} Oct 30 16:46:53.103789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:53.103395 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:46:53.152417 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:53.152355 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" podStartSLOduration=1.794642683 podStartE2EDuration="2.152339145s" totalImagesPullingTime="357.696462ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:46:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:46:53.150383493 +0000 UTC m=+1522.300664483" watchObservedRunningTime="2026-10-30 16:46:53.152339145 +0000 UTC m=+1522.302620134" Oct 30 16:46:53.285657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:53.280340 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" start-of-body= Oct 30 16:46:53.285657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:53.280400 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.110:15021/healthz/ready\": dial tcp 10.134.0.110:15021: connect: connection refused" Oct 30 16:46:54.284942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:54.284904 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:54.285455 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:54.285290 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:54.286027 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:54.286006 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:46:54.604388 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:46:54.604295 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:46:59.118018 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:46:59.117989 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:47:17.176375 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.176328 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn"] Oct 30 16:47:17.181085 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.181064 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.208559 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.208531 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn"] Oct 30 16:47:17.213941 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.213912 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-00fd1a-serving-cert\"" Oct 30 16:47:17.308573 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.308540 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzqdp\" (UniqueName: \"kubernetes.io/projected/0cc052b6-68f0-41f0-8520-0b15744583dc-kube-api-access-jzqdp\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.308760 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.308580 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-maas-api-tls\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.308760 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.308715 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc052b6-68f0-41f0-8520-0b15744583dc-openshift-service-ca\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.308840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.308772 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-metrics-certs\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.411178 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.411138 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzqdp\" (UniqueName: \"kubernetes.io/projected/0cc052b6-68f0-41f0-8520-0b15744583dc-kube-api-access-jzqdp\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.411390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.411203 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-maas-api-tls\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.411390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.411277 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc052b6-68f0-41f0-8520-0b15744583dc-openshift-service-ca\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.411390 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:47:17.411319 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-serving-cert: secret "maas-api-e2e-api-a-00fd1a-serving-cert" not found Oct 30 16:47:17.411390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.411363 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-metrics-certs\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.411390 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:47:17.411389 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-maas-api-tls podName:0cc052b6-68f0-41f0-8520-0b15744583dc nodeName:}" failed. No retries permitted until 2026-10-30 16:47:17.911365752 +0000 UTC m=+1547.061646721 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-maas-api-tls") pod "maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" (UID: "0cc052b6-68f0-41f0-8520-0b15744583dc") : secret "maas-api-e2e-api-a-00fd1a-serving-cert" not found Oct 30 16:47:17.412576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.412545 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc052b6-68f0-41f0-8520-0b15744583dc-openshift-service-ca\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.414849 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.414823 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-metrics-certs\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.434553 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.434484 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzqdp\" (UniqueName: \"kubernetes.io/projected/0cc052b6-68f0-41f0-8520-0b15744583dc-kube-api-access-jzqdp\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.919564 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.919530 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-maas-api-tls\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:17.922558 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:17.922522 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-maas-api-tls\") pod \"maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:18.097620 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:18.097567 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:18.441685 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:18.438569 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn"] Oct 30 16:47:18.502769 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:18.502697 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:47:19.423629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:19.423567 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" event={"ID":"0cc052b6-68f0-41f0-8520-0b15744583dc","Type":"ContainerStarted","Data":"b0bdcccd80bb7ade326126a72645b6246fced1d6fe931d2fdaaefc92057b8dfc"} Oct 30 16:47:19.424655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:19.424626 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" event={"ID":"0cc052b6-68f0-41f0-8520-0b15744583dc","Type":"ContainerStarted","Data":"b0470c838dd7d0b76272f68f82880f7972591c5948fe0320c3f8d03b8ba90b75"} Oct 30 16:47:19.424856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:19.424840 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:19.470815 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:19.470750 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" podStartSLOduration=2.020836709 podStartE2EDuration="2.470730649s" totalImagesPullingTime="449.89394ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:47:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:47:19.467822157 +0000 UTC m=+1548.618103147" watchObservedRunningTime="2026-10-30 16:47:19.470730649 +0000 UTC m=+1548.621011649" Oct 30 16:47:24.602431 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:47:24.602392 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:47:25.436527 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:47:25.436499 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:47:54.606763 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:47:54.606722 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:48:12.991010 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:12.990973 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf"] Oct 30 16:48:12.991557 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:12.991250 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" podUID="58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" containerName="maas-api" containerID="cri-o://c1f6e0cf5ad63907957042d651ea0b363f89a4df9c0739578a4b00a540fa96ab" gracePeriod=30 Oct 30 16:48:13.467261 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:13.454901 6400 generic.go:356] "Generic (PLEG): container finished" podID="58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" containerID="c1f6e0cf5ad63907957042d651ea0b363f89a4df9c0739578a4b00a540fa96ab" exitCode=0 Oct 30 16:48:13.467261 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:13.455073 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" event={"ID":"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb","Type":"ContainerDied","Data":"c1f6e0cf5ad63907957042d651ea0b363f89a4df9c0739578a4b00a540fa96ab"} Oct 30 16:48:13.995108 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:13.995069 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:48:14.061151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.061109 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-maas-api-tls\") pod \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " Oct 30 16:48:14.061335 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.061249 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-kube-api-access-sqs77\" (UniqueName: \"kubernetes.io/projected/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-kube-api-access-sqs77\") pod \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " Oct 30 16:48:14.061443 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.061423 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-openshift-service-ca\") pod \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " Oct 30 16:48:14.061712 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.061688 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-metrics-certs\") pod \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\" (UID: \"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb\") " Oct 30 16:48:14.062390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.062363 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-openshift-service-ca" pod "58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" (UID: "58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:14.064538 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.064495 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-maas-api-tls" pod "58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" (UID: "58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:48:14.065760 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.065700 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-kube-api-access-sqs77" pod "58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" (UID: "58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb"). InnerVolumeSpecName "kube-api-access-sqs77". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:14.065975 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.065942 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.068237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.068211 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-metrics-certs" pod "58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" (UID: "58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:48:14.169998 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.169919 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqs77\" (UniqueName: \"kubernetes.io/projected/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-kube-api-access-sqs77\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.169998 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.169948 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.169998 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.169957 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:14.464549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.464468 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" Oct 30 16:48:14.464549 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.464506 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" event={"ID":"58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb","Type":"ContainerDied","Data":"b87864f0f3d6bf3f5b085af645247182bf2860b74c981ced08d9b3d8df9f7dab"} Oct 30 16:48:14.464840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.464578 6400 scope.go:122] "RemoveContainer" containerID="c1f6e0cf5ad63907957042d651ea0b363f89a4df9c0739578a4b00a540fa96ab" Oct 30 16:48:14.516983 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.516955 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf"] Oct 30 16:48:14.521128 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.521102 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf"] Oct 30 16:48:14.969405 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:14.969372 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" path="/var/lib/kubelet/pods/58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb/volumes" Oct 30 16:48:16.519471 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:16.519424 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-b8af660d-6697cd894d-fwzgf" podUID="58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.100:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 16:48:20.387701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:20.387667 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh"] Oct 30 16:48:20.388111 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:20.387859 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" containerID="cri-o://b72c805285238641028db505aa3fe2aa9edbf70607c90e19a41d215305af67e8" gracePeriod=30 Oct 30 16:48:20.400944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:20.400911 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws"] Oct 30 16:48:20.401134 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:20.401113 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" podUID="a965f27d-01a1-4d25-9331-91fe73051186" containerName="istio-proxy" containerID="cri-o://3086068694d0be0ba4b5014c15bdcf29e668b00559ebcd5ee1a0789ea57d7406" gracePeriod=30 Oct 30 16:48:24.607146 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:48:24.607105 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:48:25.616677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.616643 6400 generic.go:356] "Generic (PLEG): container finished" podID="9db06720-8b98-4659-be16-f1a3429b0a06" containerID="b72c805285238641028db505aa3fe2aa9edbf70607c90e19a41d215305af67e8" exitCode=0 Oct 30 16:48:25.617072 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.616853 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" event={"ID":"9db06720-8b98-4659-be16-f1a3429b0a06","Type":"ContainerDied","Data":"b72c805285238641028db505aa3fe2aa9edbf70607c90e19a41d215305af67e8"} Oct 30 16:48:25.618535 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.618514 6400 generic.go:356] "Generic (PLEG): container finished" podID="a965f27d-01a1-4d25-9331-91fe73051186" containerID="3086068694d0be0ba4b5014c15bdcf29e668b00559ebcd5ee1a0789ea57d7406" exitCode=0 Oct 30 16:48:25.618618 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.618563 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" event={"ID":"a965f27d-01a1-4d25-9331-91fe73051186","Type":"ContainerDied","Data":"3086068694d0be0ba4b5014c15bdcf29e668b00559ebcd5ee1a0789ea57d7406"} Oct 30 16:48:25.763839 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.763808 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:48:25.783862 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.783834 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:48:25.874558 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.874522 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a965f27d-01a1-4d25-9331-91fe73051186-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a965f27d-01a1-4d25-9331-91fe73051186-istiod-ca-cert\") pod \"a965f27d-01a1-4d25-9331-91fe73051186\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " Oct 30 16:48:25.874855 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.874830 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-certs\") pod \"a965f27d-01a1-4d25-9331-91fe73051186\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " Oct 30 16:48:25.875013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.874996 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-certs\") pod \"9db06720-8b98-4659-be16-f1a3429b0a06\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " Oct 30 16:48:25.875157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.875142 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9db06720-8b98-4659-be16-f1a3429b0a06-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db06720-8b98-4659-be16-f1a3429b0a06-istiod-ca-cert\") pod \"9db06720-8b98-4659-be16-f1a3429b0a06\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " Oct 30 16:48:25.875298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.875283 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-istio-token\" (UniqueName: \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-istio-token\") pod \"a965f27d-01a1-4d25-9331-91fe73051186\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " Oct 30 16:48:25.875432 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.875417 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-kube-api-access-v6sr5\" (UniqueName: \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-kube-api-access-v6sr5\") pod \"9db06720-8b98-4659-be16-f1a3429b0a06\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " Oct 30 16:48:25.875628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.875588 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-credential-socket\") pod \"a965f27d-01a1-4d25-9331-91fe73051186\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " Oct 30 16:48:25.875871 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.875851 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-istio-token\" (UniqueName: \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-istio-token\") pod \"9db06720-8b98-4659-be16-f1a3429b0a06\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " Oct 30 16:48:25.876094 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.876074 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-credential-socket\") pod \"9db06720-8b98-4659-be16-f1a3429b0a06\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " Oct 30 16:48:25.876245 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.876229 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9db06720-8b98-4659-be16-f1a3429b0a06-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db06720-8b98-4659-be16-f1a3429b0a06-istio-podinfo\") pod \"9db06720-8b98-4659-be16-f1a3429b0a06\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " Oct 30 16:48:25.876419 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.876403 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-socket\") pod \"a965f27d-01a1-4d25-9331-91fe73051186\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " Oct 30 16:48:25.876563 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.876547 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-envoy\") pod \"9db06720-8b98-4659-be16-f1a3429b0a06\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " Oct 30 16:48:25.876726 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.876708 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a965f27d-01a1-4d25-9331-91fe73051186-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a965f27d-01a1-4d25-9331-91fe73051186-istio-podinfo\") pod \"a965f27d-01a1-4d25-9331-91fe73051186\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " Oct 30 16:48:25.876892 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.876831 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-data\") pod \"a965f27d-01a1-4d25-9331-91fe73051186\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " Oct 30 16:48:25.877112 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.877092 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-kube-api-access-2q4t5\" (UniqueName: \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-kube-api-access-2q4t5\") pod \"a965f27d-01a1-4d25-9331-91fe73051186\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " Oct 30 16:48:25.877243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.877228 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-data\") pod \"9db06720-8b98-4659-be16-f1a3429b0a06\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " Oct 30 16:48:25.877378 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.877352 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-socket\") pod \"9db06720-8b98-4659-be16-f1a3429b0a06\" (UID: \"9db06720-8b98-4659-be16-f1a3429b0a06\") " Oct 30 16:48:25.877490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.877406 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-envoy\") pod \"a965f27d-01a1-4d25-9331-91fe73051186\" (UID: \"a965f27d-01a1-4d25-9331-91fe73051186\") " Oct 30 16:48:25.878676 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.878636 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-credential-socket" pod "9db06720-8b98-4659-be16-f1a3429b0a06" (UID: "9db06720-8b98-4659-be16-f1a3429b0a06"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:25.878971 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.878948 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9db06720-8b98-4659-be16-f1a3429b0a06-istiod-ca-cert" pod "9db06720-8b98-4659-be16-f1a3429b0a06" (UID: "9db06720-8b98-4659-be16-f1a3429b0a06"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:25.879558 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.879513 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-socket" pod "a965f27d-01a1-4d25-9331-91fe73051186" (UID: "a965f27d-01a1-4d25-9331-91fe73051186"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:25.879965 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.879943 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a965f27d-01a1-4d25-9331-91fe73051186-istiod-ca-cert" pod "a965f27d-01a1-4d25-9331-91fe73051186" (UID: "a965f27d-01a1-4d25-9331-91fe73051186"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:48:25.880398 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.880377 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-certs" pod "a965f27d-01a1-4d25-9331-91fe73051186" (UID: "a965f27d-01a1-4d25-9331-91fe73051186"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:25.880722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.880701 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-certs" pod "9db06720-8b98-4659-be16-f1a3429b0a06" (UID: "9db06720-8b98-4659-be16-f1a3429b0a06"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:25.881040 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.881021 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-credential-socket" pod "a965f27d-01a1-4d25-9331-91fe73051186" (UID: "a965f27d-01a1-4d25-9331-91fe73051186"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:25.881366 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.881347 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-data" pod "a965f27d-01a1-4d25-9331-91fe73051186" (UID: "a965f27d-01a1-4d25-9331-91fe73051186"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:25.881477 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.881415 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-data" pod "9db06720-8b98-4659-be16-f1a3429b0a06" (UID: "9db06720-8b98-4659-be16-f1a3429b0a06"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:25.881799 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.881778 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-socket" pod "9db06720-8b98-4659-be16-f1a3429b0a06" (UID: "9db06720-8b98-4659-be16-f1a3429b0a06"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:25.882991 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.882952 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9db06720-8b98-4659-be16-f1a3429b0a06-istio-podinfo" pod "9db06720-8b98-4659-be16-f1a3429b0a06" (UID: "9db06720-8b98-4659-be16-f1a3429b0a06"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:48:25.885523 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.885496 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-envoy" pod "a965f27d-01a1-4d25-9331-91fe73051186" (UID: "a965f27d-01a1-4d25-9331-91fe73051186"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:25.889881 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.889807 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-kube-api-access-2q4t5" pod "a965f27d-01a1-4d25-9331-91fe73051186" (UID: "a965f27d-01a1-4d25-9331-91fe73051186"). InnerVolumeSpecName "kube-api-access-2q4t5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:25.889881 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.889849 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-istio-token" pod "9db06720-8b98-4659-be16-f1a3429b0a06" (UID: "9db06720-8b98-4659-be16-f1a3429b0a06"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:25.890198 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.889999 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-istio-token" pod "a965f27d-01a1-4d25-9331-91fe73051186" (UID: "a965f27d-01a1-4d25-9331-91fe73051186"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:25.890820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.890792 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-envoy" pod "9db06720-8b98-4659-be16-f1a3429b0a06" (UID: "9db06720-8b98-4659-be16-f1a3429b0a06"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:48:25.891752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.891732 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a965f27d-01a1-4d25-9331-91fe73051186-istio-podinfo" pod "a965f27d-01a1-4d25-9331-91fe73051186" (UID: "a965f27d-01a1-4d25-9331-91fe73051186"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:48:25.891834 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.891788 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-kube-api-access-v6sr5" pod "9db06720-8b98-4659-be16-f1a3429b0a06" (UID: "9db06720-8b98-4659-be16-f1a3429b0a06"). InnerVolumeSpecName "kube-api-access-v6sr5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:48:25.985159 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985123 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a965f27d-01a1-4d25-9331-91fe73051186-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985159 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985162 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985174 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985187 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9db06720-8b98-4659-be16-f1a3429b0a06-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985198 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985212 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6sr5\" (UniqueName: \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-kube-api-access-v6sr5\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985226 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985240 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9db06720-8b98-4659-be16-f1a3429b0a06-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985251 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985263 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9db06720-8b98-4659-be16-f1a3429b0a06-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985276 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985288 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985300 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a965f27d-01a1-4d25-9331-91fe73051186-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985310 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985323 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q4t5\" (UniqueName: \"kubernetes.io/projected/a965f27d-01a1-4d25-9331-91fe73051186-kube-api-access-2q4t5\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985334 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985348 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9db06720-8b98-4659-be16-f1a3429b0a06-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:25.985492 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:25.985361 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a965f27d-01a1-4d25-9331-91fe73051186-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:48:26.631706 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.631682 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" Oct 30 16:48:26.632195 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.631744 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh" event={"ID":"9db06720-8b98-4659-be16-f1a3429b0a06","Type":"ContainerDied","Data":"0728d2c8f8f03cb51bde9079c77eee3805ab3627bd2f61472ac0a317b01ef21b"} Oct 30 16:48:26.632195 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.631815 6400 scope.go:122] "RemoveContainer" containerID="b72c805285238641028db505aa3fe2aa9edbf70607c90e19a41d215305af67e8" Oct 30 16:48:26.633774 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.633753 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" event={"ID":"a965f27d-01a1-4d25-9331-91fe73051186","Type":"ContainerDied","Data":"2f49f394b1c48574f829fb35590debf9bab27d361e26d0af38a774009c794eef"} Oct 30 16:48:26.633874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.633790 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws" Oct 30 16:48:26.643738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.643718 6400 scope.go:122] "RemoveContainer" containerID="3086068694d0be0ba4b5014c15bdcf29e668b00559ebcd5ee1a0789ea57d7406" Oct 30 16:48:26.697972 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.697889 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws"] Oct 30 16:48:26.701850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.701826 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-pp7ws"] Oct 30 16:48:26.731674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.731639 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh"] Oct 30 16:48:26.740965 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.740934 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-sp9lh"] Oct 30 16:48:26.979487 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.979392 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" path="/var/lib/kubelet/pods/9db06720-8b98-4659-be16-f1a3429b0a06/volumes" Oct 30 16:48:26.980043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:48:26.980021 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a965f27d-01a1-4d25-9331-91fe73051186" path="/var/lib/kubelet/pods/a965f27d-01a1-4d25-9331-91fe73051186/volumes" Oct 30 16:48:54.605854 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:48:54.605815 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:49:24.601083 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:49:24.601047 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:49:54.607247 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:49:54.607204 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:50:03.846825 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:03.846789 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b"] Oct 30 16:50:03.847326 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:03.847039 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" podUID="715f3a38-e8db-443f-9490-3ad2b99370ea" containerName="maas-api" containerID="cri-o://94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8" gracePeriod=30 Oct 30 16:50:04.399630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.395422 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:50:04.579820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.576545 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-maas-api-tls\") pod \"715f3a38-e8db-443f-9490-3ad2b99370ea\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " Oct 30 16:50:04.579820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.576670 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/715f3a38-e8db-443f-9490-3ad2b99370ea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715f3a38-e8db-443f-9490-3ad2b99370ea-openshift-service-ca\") pod \"715f3a38-e8db-443f-9490-3ad2b99370ea\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " Oct 30 16:50:04.579820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.576860 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/715f3a38-e8db-443f-9490-3ad2b99370ea-kube-api-access-bkzjg\" (UniqueName: \"kubernetes.io/projected/715f3a38-e8db-443f-9490-3ad2b99370ea-kube-api-access-bkzjg\") pod \"715f3a38-e8db-443f-9490-3ad2b99370ea\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " Oct 30 16:50:04.579820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.576977 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-metrics-certs\") pod \"715f3a38-e8db-443f-9490-3ad2b99370ea\" (UID: \"715f3a38-e8db-443f-9490-3ad2b99370ea\") " Oct 30 16:50:04.587144 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.587096 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/715f3a38-e8db-443f-9490-3ad2b99370ea-openshift-service-ca" pod "715f3a38-e8db-443f-9490-3ad2b99370ea" (UID: "715f3a38-e8db-443f-9490-3ad2b99370ea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:04.615836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.615725 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-maas-api-tls" pod "715f3a38-e8db-443f-9490-3ad2b99370ea" (UID: "715f3a38-e8db-443f-9490-3ad2b99370ea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:04.617912 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.617450 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/715f3a38-e8db-443f-9490-3ad2b99370ea-kube-api-access-bkzjg" pod "715f3a38-e8db-443f-9490-3ad2b99370ea" (UID: "715f3a38-e8db-443f-9490-3ad2b99370ea"). InnerVolumeSpecName "kube-api-access-bkzjg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:04.646771 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.646735 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-metrics-certs" pod "715f3a38-e8db-443f-9490-3ad2b99370ea" (UID: "715f3a38-e8db-443f-9490-3ad2b99370ea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:04.703720 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.694451 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.703720 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.694486 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/715f3a38-e8db-443f-9490-3ad2b99370ea-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.703720 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.694501 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/715f3a38-e8db-443f-9490-3ad2b99370ea-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.703720 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.694517 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkzjg\" (UniqueName: \"kubernetes.io/projected/715f3a38-e8db-443f-9490-3ad2b99370ea-kube-api-access-bkzjg\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:04.763248 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.763206 6400 generic.go:356] "Generic (PLEG): container finished" podID="715f3a38-e8db-443f-9490-3ad2b99370ea" containerID="94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8" exitCode=0 Oct 30 16:50:04.763452 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.763414 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" event={"ID":"715f3a38-e8db-443f-9490-3ad2b99370ea","Type":"ContainerDied","Data":"94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8"} Oct 30 16:50:04.763514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.763471 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" event={"ID":"715f3a38-e8db-443f-9490-3ad2b99370ea","Type":"ContainerDied","Data":"89b2af90648425e88461b2d6338851d0ddf87b0c9f134079de3dcd0a72189dfa"} Oct 30 16:50:04.763514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.763502 6400 scope.go:122] "RemoveContainer" containerID="94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8" Oct 30 16:50:04.763734 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.763711 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b" Oct 30 16:50:04.782071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.782044 6400 scope.go:122] "RemoveContainer" containerID="94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8" Oct 30 16:50:04.782422 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:50:04.782398 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8\": container with ID starting with 94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8 not found: ID does not exist" containerID="94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8" Oct 30 16:50:04.782504 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.782435 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8"} err="failed to get container status \"94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8\": rpc error: code = NotFound desc = could not find container \"94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8\": container with ID starting with 94fd456d1ea1961403f7fe9a6c33de60cb535f22c99ddce4dd06cb59f6caeaa8 not found: ID does not exist" Oct 30 16:50:04.820823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.820787 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b"] Oct 30 16:50:04.826285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.826252 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-968df4-6f49dbf945-nd49b"] Oct 30 16:50:04.968930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:04.968892 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="715f3a38-e8db-443f-9490-3ad2b99370ea" path="/var/lib/kubelet/pods/715f3a38-e8db-443f-9490-3ad2b99370ea/volumes" Oct 30 16:50:11.695262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:11.695165 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4"] Oct 30 16:50:11.695809 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:11.695491 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" podUID="d0d8af8b-c70a-4a25-bd13-577b099058ec" containerName="maas-api" containerID="cri-o://86ec40e3392e9aa7e58971c203432cb86b765c1f3e57f206cca9fc8aaacc734b" gracePeriod=30 Oct 30 16:50:11.868238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:11.868190 6400 generic.go:356] "Generic (PLEG): container finished" podID="d0d8af8b-c70a-4a25-bd13-577b099058ec" containerID="86ec40e3392e9aa7e58971c203432cb86b765c1f3e57f206cca9fc8aaacc734b" exitCode=0 Oct 30 16:50:11.868444 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:11.868426 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" event={"ID":"d0d8af8b-c70a-4a25-bd13-577b099058ec","Type":"ContainerDied","Data":"86ec40e3392e9aa7e58971c203432cb86b765c1f3e57f206cca9fc8aaacc734b"} Oct 30 16:50:12.367004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.366954 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:50:12.462741 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.458127 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-metrics-certs\") pod \"d0d8af8b-c70a-4a25-bd13-577b099058ec\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " Oct 30 16:50:12.462741 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.458229 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0d8af8b-c70a-4a25-bd13-577b099058ec-kube-api-access-gnldf\" (UniqueName: \"kubernetes.io/projected/d0d8af8b-c70a-4a25-bd13-577b099058ec-kube-api-access-gnldf\") pod \"d0d8af8b-c70a-4a25-bd13-577b099058ec\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " Oct 30 16:50:12.462741 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.458464 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0d8af8b-c70a-4a25-bd13-577b099058ec-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d8af8b-c70a-4a25-bd13-577b099058ec-openshift-service-ca\") pod \"d0d8af8b-c70a-4a25-bd13-577b099058ec\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " Oct 30 16:50:12.462741 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.458553 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-maas-api-tls\") pod \"d0d8af8b-c70a-4a25-bd13-577b099058ec\" (UID: \"d0d8af8b-c70a-4a25-bd13-577b099058ec\") " Oct 30 16:50:12.477802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.463566 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-maas-api-tls" pod "d0d8af8b-c70a-4a25-bd13-577b099058ec" (UID: "d0d8af8b-c70a-4a25-bd13-577b099058ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:12.477802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.466917 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-metrics-certs" pod "d0d8af8b-c70a-4a25-bd13-577b099058ec" (UID: "d0d8af8b-c70a-4a25-bd13-577b099058ec"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:50:12.477802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.472782 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0d8af8b-c70a-4a25-bd13-577b099058ec-openshift-service-ca" pod "d0d8af8b-c70a-4a25-bd13-577b099058ec" (UID: "d0d8af8b-c70a-4a25-bd13-577b099058ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:12.495292 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.492677 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0d8af8b-c70a-4a25-bd13-577b099058ec-kube-api-access-gnldf" pod "d0d8af8b-c70a-4a25-bd13-577b099058ec" (UID: "d0d8af8b-c70a-4a25-bd13-577b099058ec"). InnerVolumeSpecName "kube-api-access-gnldf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:12.561642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.561594 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0d8af8b-c70a-4a25-bd13-577b099058ec-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.561642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.561637 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.561642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.561647 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0d8af8b-c70a-4a25-bd13-577b099058ec-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.561890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.561657 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnldf\" (UniqueName: \"kubernetes.io/projected/d0d8af8b-c70a-4a25-bd13-577b099058ec-kube-api-access-gnldf\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:12.922461 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.922410 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" event={"ID":"d0d8af8b-c70a-4a25-bd13-577b099058ec","Type":"ContainerDied","Data":"cebaac585d3463e32f341882a36c38c727a9c07ffe27764c50615493e3241b61"} Oct 30 16:50:12.923069 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.923048 6400 scope.go:122] "RemoveContainer" containerID="86ec40e3392e9aa7e58971c203432cb86b765c1f3e57f206cca9fc8aaacc734b" Oct 30 16:50:12.923451 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:12.923427 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4" Oct 30 16:50:13.182631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:13.180275 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4"] Oct 30 16:50:13.190232 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:13.189014 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-b315d4-5c4b7cd7f7-bl2p4"] Oct 30 16:50:15.060629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:15.051900 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0d8af8b-c70a-4a25-bd13-577b099058ec" path="/var/lib/kubelet/pods/d0d8af8b-c70a-4a25-bd13-577b099058ec/volumes" Oct 30 16:50:20.835572 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:20.835536 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h"] Oct 30 16:50:20.836049 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:20.835801 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerName="istio-proxy" containerID="cri-o://832bf20a6d6d9a20e3f151d04e14d300adf3fc34cccb6a9be05aeec255fe06f1" gracePeriod=30 Oct 30 16:50:20.852353 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:20.852324 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw"] Oct 30 16:50:20.852533 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:20.852515 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerName="istio-proxy" containerID="cri-o://04f0cdd2cdd154abdf3fcf11e646ccb3500528f90fd2796d0590f125cd581cf5" gracePeriod=30 Oct 30 16:50:21.129140 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:21.129058 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd"] Oct 30 16:50:21.129297 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:21.129250 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" containerID="cri-o://1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff" gracePeriod=30 Oct 30 16:50:21.375262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:21.375227 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj"] Oct 30 16:50:21.375458 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:21.375436 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" containerID="cri-o://e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d" gracePeriod=30 Oct 30 16:50:24.835717 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:50:24.835674 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:50:26.134576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.109804 6400 generic.go:356] "Generic (PLEG): container finished" podID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerID="04f0cdd2cdd154abdf3fcf11e646ccb3500528f90fd2796d0590f125cd581cf5" exitCode=0 Oct 30 16:50:26.134576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.109849 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" event={"ID":"59f491b6-5fc5-4372-9a50-d60c3d5d6774","Type":"ContainerDied","Data":"04f0cdd2cdd154abdf3fcf11e646ccb3500528f90fd2796d0590f125cd581cf5"} Oct 30 16:50:26.134576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.111817 6400 generic.go:356] "Generic (PLEG): container finished" podID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerID="832bf20a6d6d9a20e3f151d04e14d300adf3fc34cccb6a9be05aeec255fe06f1" exitCode=0 Oct 30 16:50:26.134576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.111986 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" event={"ID":"e5bc8d59-5d70-4dc7-954b-6c7310187f19","Type":"ContainerDied","Data":"832bf20a6d6d9a20e3f151d04e14d300adf3fc34cccb6a9be05aeec255fe06f1"} Oct 30 16:50:26.229165 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.229140 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:50:26.243145 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.243120 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:50:26.385771 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.385674 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-socket\") pod \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " Oct 30 16:50:26.385771 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.385753 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istiod-ca-cert\") pod \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " Oct 30 16:50:26.386009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.385831 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-token\" (UniqueName: \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-token\") pod \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " Oct 30 16:50:26.386009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.385859 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istiod-ca-cert\") pod \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " Oct 30 16:50:26.386009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.385935 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-envoy\") pod \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " Oct 30 16:50:26.386009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.385978 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-podinfo\") pod \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386080 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-kube-api-access-n5lr2\" (UniqueName: \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-kube-api-access-n5lr2\") pod \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386113 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-token\" (UniqueName: \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-token\") pod \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386150 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-certs\") pod \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386197 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-credential-socket\") pod \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386260 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-podinfo\") pod \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386289 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-data\") pod \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386359 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-socket\") pod \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386464 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-credential-socket\") pod \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386621 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-certs\") pod \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386655 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-data\") pod \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386694 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-kube-api-access-25mzr\" (UniqueName: \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-kube-api-access-25mzr\") pod \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\" (UID: \"59f491b6-5fc5-4372-9a50-d60c3d5d6774\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.386752 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-envoy\") pod \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\" (UID: \"e5bc8d59-5d70-4dc7-954b-6c7310187f19\") " Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.387064 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-certs" pod "59f491b6-5fc5-4372-9a50-d60c3d5d6774" (UID: "59f491b6-5fc5-4372-9a50-d60c3d5d6774"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.387883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.387277 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-credential-socket" pod "59f491b6-5fc5-4372-9a50-d60c3d5d6774" (UID: "59f491b6-5fc5-4372-9a50-d60c3d5d6774"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.388821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.388243 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-credential-socket" pod "e5bc8d59-5d70-4dc7-954b-6c7310187f19" (UID: "e5bc8d59-5d70-4dc7-954b-6c7310187f19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.388821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.388778 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.388821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.388801 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.388821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.388816 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.389043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.389030 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-data" pod "e5bc8d59-5d70-4dc7-954b-6c7310187f19" (UID: "e5bc8d59-5d70-4dc7-954b-6c7310187f19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.389899 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.389327 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-socket" pod "59f491b6-5fc5-4372-9a50-d60c3d5d6774" (UID: "59f491b6-5fc5-4372-9a50-d60c3d5d6774"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.389899 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.389339 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-data" pod "59f491b6-5fc5-4372-9a50-d60c3d5d6774" (UID: "59f491b6-5fc5-4372-9a50-d60c3d5d6774"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.389899 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.389518 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-certs" pod "e5bc8d59-5d70-4dc7-954b-6c7310187f19" (UID: "e5bc8d59-5d70-4dc7-954b-6c7310187f19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.390131 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.389900 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-socket" pod "e5bc8d59-5d70-4dc7-954b-6c7310187f19" (UID: "e5bc8d59-5d70-4dc7-954b-6c7310187f19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.390131 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.390027 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istiod-ca-cert" pod "e5bc8d59-5d70-4dc7-954b-6c7310187f19" (UID: "e5bc8d59-5d70-4dc7-954b-6c7310187f19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:26.390729 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.390681 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istiod-ca-cert" pod "59f491b6-5fc5-4372-9a50-d60c3d5d6774" (UID: "59f491b6-5fc5-4372-9a50-d60c3d5d6774"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:26.390962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.390937 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-envoy" pod "e5bc8d59-5d70-4dc7-954b-6c7310187f19" (UID: "e5bc8d59-5d70-4dc7-954b-6c7310187f19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.391957 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.391929 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-envoy" pod "59f491b6-5fc5-4372-9a50-d60c3d5d6774" (UID: "59f491b6-5fc5-4372-9a50-d60c3d5d6774"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.395918 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.395781 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-podinfo" pod "59f491b6-5fc5-4372-9a50-d60c3d5d6774" (UID: "59f491b6-5fc5-4372-9a50-d60c3d5d6774"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:26.395918 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.395814 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-kube-api-access-25mzr" pod "59f491b6-5fc5-4372-9a50-d60c3d5d6774" (UID: "59f491b6-5fc5-4372-9a50-d60c3d5d6774"). InnerVolumeSpecName "kube-api-access-25mzr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.395918 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.395881 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-token" pod "e5bc8d59-5d70-4dc7-954b-6c7310187f19" (UID: "e5bc8d59-5d70-4dc7-954b-6c7310187f19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.396295 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.396177 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-token" pod "59f491b6-5fc5-4372-9a50-d60c3d5d6774" (UID: "59f491b6-5fc5-4372-9a50-d60c3d5d6774"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.397636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.397573 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-podinfo" pod "e5bc8d59-5d70-4dc7-954b-6c7310187f19" (UID: "e5bc8d59-5d70-4dc7-954b-6c7310187f19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:26.399284 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.399249 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-kube-api-access-n5lr2" pod "e5bc8d59-5d70-4dc7-954b-6c7310187f19" (UID: "e5bc8d59-5d70-4dc7-954b-6c7310187f19"). InnerVolumeSpecName "kube-api-access-n5lr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.488913 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.488889 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:50:26.490154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490094 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5lr2\" (UniqueName: \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-kube-api-access-n5lr2\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490112 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490121 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490129 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490139 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490148 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490156 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490164 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25mzr\" (UniqueName: \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-kube-api-access-25mzr\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490172 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490180 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490187 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490197 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490204 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490214 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59f491b6-5fc5-4372-9a50-d60c3d5d6774-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.490382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.490221 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5bc8d59-5d70-4dc7-954b-6c7310187f19-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.598590 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.598552 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-data\") pod \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " Oct 30 16:50:26.598590 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.598626 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-kube-api-access-pt2pl\" (UniqueName: \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-kube-api-access-pt2pl\") pod \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " Oct 30 16:50:26.599019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.598721 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istiod-ca-cert\") pod \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " Oct 30 16:50:26.599019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.598761 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-certs\") pod \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " Oct 30 16:50:26.599019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.598803 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-socket\") pod \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " Oct 30 16:50:26.599019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.598896 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-envoy\") pod \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " Oct 30 16:50:26.599019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.598926 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-credential-socket\") pod \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " Oct 30 16:50:26.599019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.598958 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-podinfo\") pod \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " Oct 30 16:50:26.599503 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.599086 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-token\" (UniqueName: \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-token\") pod \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\" (UID: \"89880f7f-514c-4cbf-b623-c0ab7c16d8a7\") " Oct 30 16:50:26.600527 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.600496 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-certs" pod "89880f7f-514c-4cbf-b623-c0ab7c16d8a7" (UID: "89880f7f-514c-4cbf-b623-c0ab7c16d8a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.600756 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.600726 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-data" pod "89880f7f-514c-4cbf-b623-c0ab7c16d8a7" (UID: "89880f7f-514c-4cbf-b623-c0ab7c16d8a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.601800 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.601774 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istiod-ca-cert" pod "89880f7f-514c-4cbf-b623-c0ab7c16d8a7" (UID: "89880f7f-514c-4cbf-b623-c0ab7c16d8a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:26.601976 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.601959 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-credential-socket" pod "89880f7f-514c-4cbf-b623-c0ab7c16d8a7" (UID: "89880f7f-514c-4cbf-b623-c0ab7c16d8a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.602244 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.602213 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-socket" pod "89880f7f-514c-4cbf-b623-c0ab7c16d8a7" (UID: "89880f7f-514c-4cbf-b623-c0ab7c16d8a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.603363 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.603337 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-kube-api-access-pt2pl" pod "89880f7f-514c-4cbf-b623-c0ab7c16d8a7" (UID: "89880f7f-514c-4cbf-b623-c0ab7c16d8a7"). InnerVolumeSpecName "kube-api-access-pt2pl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.606131 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.606094 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-podinfo" pod "89880f7f-514c-4cbf-b623-c0ab7c16d8a7" (UID: "89880f7f-514c-4cbf-b623-c0ab7c16d8a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:26.606243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.606143 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-token" pod "89880f7f-514c-4cbf-b623-c0ab7c16d8a7" (UID: "89880f7f-514c-4cbf-b623-c0ab7c16d8a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.613346 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.613314 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-envoy" pod "89880f7f-514c-4cbf-b623-c0ab7c16d8a7" (UID: "89880f7f-514c-4cbf-b623-c0ab7c16d8a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.686834 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.686809 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:50:26.701196 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.701168 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.701196 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.701194 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt2pl\" (UniqueName: \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-kube-api-access-pt2pl\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.701196 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.701205 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.701391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.701225 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.701391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.701233 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.701391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.701241 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.701391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.701249 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.701391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.701258 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.701391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.701267 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/89880f7f-514c-4cbf-b623-c0ab7c16d8a7-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.802003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.801963 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-socket\") pod \"07bb001a-9638-4cd5-a836-fb2e2cfad946\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " Oct 30 16:50:26.802230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802089 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07bb001a-9638-4cd5-a836-fb2e2cfad946-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bb001a-9638-4cd5-a836-fb2e2cfad946-istiod-ca-cert\") pod \"07bb001a-9638-4cd5-a836-fb2e2cfad946\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " Oct 30 16:50:26.802230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802137 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-token\" (UniqueName: \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-token\") pod \"07bb001a-9638-4cd5-a836-fb2e2cfad946\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " Oct 30 16:50:26.802230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802200 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-podinfo\") pod \"07bb001a-9638-4cd5-a836-fb2e2cfad946\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " Oct 30 16:50:26.802372 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802260 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-socket" pod "07bb001a-9638-4cd5-a836-fb2e2cfad946" (UID: "07bb001a-9638-4cd5-a836-fb2e2cfad946"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.802372 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802284 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-certs\") pod \"07bb001a-9638-4cd5-a836-fb2e2cfad946\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " Oct 30 16:50:26.802372 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802334 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-kube-api-access-v2xwv\" (UniqueName: \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-kube-api-access-v2xwv\") pod \"07bb001a-9638-4cd5-a836-fb2e2cfad946\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " Oct 30 16:50:26.802515 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802500 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-credential-socket\") pod \"07bb001a-9638-4cd5-a836-fb2e2cfad946\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " Oct 30 16:50:26.802560 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802526 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-data\") pod \"07bb001a-9638-4cd5-a836-fb2e2cfad946\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " Oct 30 16:50:26.802653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802559 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-envoy\") pod \"07bb001a-9638-4cd5-a836-fb2e2cfad946\" (UID: \"07bb001a-9638-4cd5-a836-fb2e2cfad946\") " Oct 30 16:50:26.802736 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802685 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07bb001a-9638-4cd5-a836-fb2e2cfad946-istiod-ca-cert" pod "07bb001a-9638-4cd5-a836-fb2e2cfad946" (UID: "07bb001a-9638-4cd5-a836-fb2e2cfad946"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:50:26.802826 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802799 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-certs" pod "07bb001a-9638-4cd5-a836-fb2e2cfad946" (UID: "07bb001a-9638-4cd5-a836-fb2e2cfad946"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.803070 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.802976 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-credential-socket" pod "07bb001a-9638-4cd5-a836-fb2e2cfad946" (UID: "07bb001a-9638-4cd5-a836-fb2e2cfad946"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.803551 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.803527 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07bb001a-9638-4cd5-a836-fb2e2cfad946-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.803677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.803556 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.803677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.803629 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.803677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.803645 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.803801 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.803676 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-data" pod "07bb001a-9638-4cd5-a836-fb2e2cfad946" (UID: "07bb001a-9638-4cd5-a836-fb2e2cfad946"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.805122 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.805087 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-token" pod "07bb001a-9638-4cd5-a836-fb2e2cfad946" (UID: "07bb001a-9638-4cd5-a836-fb2e2cfad946"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.805457 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.805432 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-podinfo" pod "07bb001a-9638-4cd5-a836-fb2e2cfad946" (UID: "07bb001a-9638-4cd5-a836-fb2e2cfad946"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:50:26.805815 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.805789 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-envoy" pod "07bb001a-9638-4cd5-a836-fb2e2cfad946" (UID: "07bb001a-9638-4cd5-a836-fb2e2cfad946"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:50:26.805966 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.805947 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-kube-api-access-v2xwv" pod "07bb001a-9638-4cd5-a836-fb2e2cfad946" (UID: "07bb001a-9638-4cd5-a836-fb2e2cfad946"). InnerVolumeSpecName "kube-api-access-v2xwv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:26.905224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.905191 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.905224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.905219 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.905224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.905230 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2xwv\" (UniqueName: \"kubernetes.io/projected/07bb001a-9638-4cd5-a836-fb2e2cfad946-kube-api-access-v2xwv\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.905509 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.905241 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:26.905509 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:26.905250 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07bb001a-9638-4cd5-a836-fb2e2cfad946-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:27.132614 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.132577 6400 generic.go:356] "Generic (PLEG): container finished" podID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerID="e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d" exitCode=0 Oct 30 16:50:27.132821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.132680 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" Oct 30 16:50:27.132821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.132675 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" event={"ID":"07bb001a-9638-4cd5-a836-fb2e2cfad946","Type":"ContainerDied","Data":"e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d"} Oct 30 16:50:27.132821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.132767 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj" event={"ID":"07bb001a-9638-4cd5-a836-fb2e2cfad946","Type":"ContainerDied","Data":"37c841013e9a338f5a25008c6c70996b7f5270b86c7194d975b2c18ad0d087f1"} Oct 30 16:50:27.132821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.132791 6400 scope.go:122] "RemoveContainer" containerID="e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d" Oct 30 16:50:27.134333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.134310 6400 generic.go:356] "Generic (PLEG): container finished" podID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerID="1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff" exitCode=0 Oct 30 16:50:27.134443 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.134362 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" Oct 30 16:50:27.134443 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.134397 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" event={"ID":"89880f7f-514c-4cbf-b623-c0ab7c16d8a7","Type":"ContainerDied","Data":"1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff"} Oct 30 16:50:27.134443 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.134436 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd" event={"ID":"89880f7f-514c-4cbf-b623-c0ab7c16d8a7","Type":"ContainerDied","Data":"2fa49904de572e9d5ee1d6186d15dbfef9476785806fc7183830bbe60bafbfe5"} Oct 30 16:50:27.136080 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.136047 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" event={"ID":"59f491b6-5fc5-4372-9a50-d60c3d5d6774","Type":"ContainerDied","Data":"c1fac2e6ae0d0a8d2df9081f74b6d606e4f787b66bfad21d19d7ff5327964dea"} Oct 30 16:50:27.136450 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.136070 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw" Oct 30 16:50:27.137820 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.137800 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" Oct 30 16:50:27.137900 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.137805 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h" event={"ID":"e5bc8d59-5d70-4dc7-954b-6c7310187f19","Type":"ContainerDied","Data":"6d6daae2cd2862d00a532f9e146061dee5d59e88d1c31093d94ed2db8ec96e93"} Oct 30 16:50:27.146107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.146090 6400 scope.go:122] "RemoveContainer" containerID="e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d" Oct 30 16:50:27.146502 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:50:27.146479 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d\": container with ID starting with e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d not found: ID does not exist" containerID="e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d" Oct 30 16:50:27.146593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.146513 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d"} err="failed to get container status \"e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d\": rpc error: code = NotFound desc = could not find container \"e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d\": container with ID starting with e052a9ab75b2b76669d041bb4f8a8e566f6f833a90cceaa56cefbc29b1584a3d not found: ID does not exist" Oct 30 16:50:27.146593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.146531 6400 scope.go:122] "RemoveContainer" containerID="1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff" Oct 30 16:50:27.156840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.156824 6400 scope.go:122] "RemoveContainer" containerID="1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff" Oct 30 16:50:27.157107 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:50:27.157089 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff\": container with ID starting with 1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff not found: ID does not exist" containerID="1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff" Oct 30 16:50:27.157237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.157119 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff"} err="failed to get container status \"1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff\": rpc error: code = NotFound desc = could not find container \"1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff\": container with ID starting with 1bacc8acf0a9e1fbae8c02798bdcd20858d5e927d3bc4bb07781c9bcbdff4fff not found: ID does not exist" Oct 30 16:50:27.157237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.157141 6400 scope.go:122] "RemoveContainer" containerID="04f0cdd2cdd154abdf3fcf11e646ccb3500528f90fd2796d0590f125cd581cf5" Oct 30 16:50:27.166973 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.166957 6400 scope.go:122] "RemoveContainer" containerID="832bf20a6d6d9a20e3f151d04e14d300adf3fc34cccb6a9be05aeec255fe06f1" Oct 30 16:50:27.184243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.184218 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h"] Oct 30 16:50:27.194959 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.194937 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-j7c5h"] Oct 30 16:50:27.232295 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.232195 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj"] Oct 30 16:50:27.238584 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.238555 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-vc7lj"] Oct 30 16:50:27.266313 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.266274 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd"] Oct 30 16:50:27.272478 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.272448 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-hp2rd"] Oct 30 16:50:27.306366 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.306323 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw"] Oct 30 16:50:27.312589 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:27.312548 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-dbnnw"] Oct 30 16:50:28.969147 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:28.969111 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" path="/var/lib/kubelet/pods/07bb001a-9638-4cd5-a836-fb2e2cfad946/volumes" Oct 30 16:50:28.969661 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:28.969647 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" path="/var/lib/kubelet/pods/59f491b6-5fc5-4372-9a50-d60c3d5d6774/volumes" Oct 30 16:50:28.970095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:28.970083 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" path="/var/lib/kubelet/pods/89880f7f-514c-4cbf-b623-c0ab7c16d8a7/volumes" Oct 30 16:50:28.970872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:28.970858 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" path="/var/lib/kubelet/pods/e5bc8d59-5d70-4dc7-954b-6c7310187f19/volumes" Oct 30 16:50:31.939376 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.939341 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z"] Oct 30 16:50:31.939989 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.939974 6400 cpu_manager.go:414] "containerMap: removing container" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" Oct 30 16:50:31.940035 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.939990 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" Oct 30 16:50:31.940035 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940005 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerName="istio-proxy" Oct 30 16:50:31.940035 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940011 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerName="istio-proxy" Oct 30 16:50:31.940035 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940019 6400 cpu_manager.go:414] "containerMap: removing container" podUID="715f3a38-e8db-443f-9490-3ad2b99370ea" containerName="maas-api" Oct 30 16:50:31.940035 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940025 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="715f3a38-e8db-443f-9490-3ad2b99370ea" containerName="maas-api" Oct 30 16:50:31.940035 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940037 6400 cpu_manager.go:414] "containerMap: removing container" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerName="istio-proxy" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940041 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerName="istio-proxy" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940054 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a965f27d-01a1-4d25-9331-91fe73051186" containerName="istio-proxy" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940059 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a965f27d-01a1-4d25-9331-91fe73051186" containerName="istio-proxy" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940068 6400 cpu_manager.go:414] "containerMap: removing container" podUID="d0d8af8b-c70a-4a25-bd13-577b099058ec" containerName="maas-api" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940072 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0d8af8b-c70a-4a25-bd13-577b099058ec" containerName="maas-api" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940091 6400 cpu_manager.go:414] "containerMap: removing container" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940096 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940117 6400 cpu_manager.go:414] "containerMap: removing container" podUID="58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" containerName="maas-api" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940122 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" containerName="maas-api" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940132 6400 cpu_manager.go:414] "containerMap: removing container" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" Oct 30 16:50:31.940209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940137 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" Oct 30 16:50:31.940555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940239 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="58a45cce-f5e9-4f18-bb5f-4ea9ac30d7fb" containerName="maas-api" Oct 30 16:50:31.940555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940247 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="89880f7f-514c-4cbf-b623-c0ab7c16d8a7" containerName="istio-proxy" Oct 30 16:50:31.940555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940256 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="9db06720-8b98-4659-be16-f1a3429b0a06" containerName="istio-proxy" Oct 30 16:50:31.940555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940262 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5bc8d59-5d70-4dc7-954b-6c7310187f19" containerName="istio-proxy" Oct 30 16:50:31.940555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940269 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="715f3a38-e8db-443f-9490-3ad2b99370ea" containerName="maas-api" Oct 30 16:50:31.940555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940277 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="59f491b6-5fc5-4372-9a50-d60c3d5d6774" containerName="istio-proxy" Oct 30 16:50:31.940555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940283 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a965f27d-01a1-4d25-9331-91fe73051186" containerName="istio-proxy" Oct 30 16:50:31.940555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940287 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0d8af8b-c70a-4a25-bd13-577b099058ec" containerName="maas-api" Oct 30 16:50:31.940555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.940294 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="07bb001a-9638-4cd5-a836-fb2e2cfad946" containerName="istio-proxy" Oct 30 16:50:31.946217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.946197 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:31.974753 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.974001 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z"] Oct 30 16:50:31.977833 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.977807 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-ff1189-serving-cert\"" Oct 30 16:50:31.993444 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.993413 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-metrics-certs\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:31.993677 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.993657 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:31.993774 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.993698 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4g96\" (UniqueName: \"kubernetes.io/projected/46fba546-bf94-4362-9b30-16d0aa688d8b-kube-api-access-t4g96\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:31.993774 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:31.993731 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fba546-bf94-4362-9b30-16d0aa688d8b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:32.057857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.057826 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8962-44aa17"] Oct 30 16:50:32.062175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.062155 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-44aa17" Oct 30 16:50:32.077491 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.077469 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr4c7\"" Oct 30 16:50:32.079743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.079697 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8962-44aa17"] Oct 30 16:50:32.095866 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.095833 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-metrics-certs\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:32.096061 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.096041 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:32.096138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.096083 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4g96\" (UniqueName: \"kubernetes.io/projected/46fba546-bf94-4362-9b30-16d0aa688d8b-kube-api-access-t4g96\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:32.096138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.096109 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fba546-bf94-4362-9b30-16d0aa688d8b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:32.096944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.096921 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fba546-bf94-4362-9b30-16d0aa688d8b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:32.099495 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.099472 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-maas-api-tls\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:32.100209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.100187 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-metrics-certs\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:32.124003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.123977 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4g96\" (UniqueName: \"kubernetes.io/projected/46fba546-bf94-4362-9b30-16d0aa688d8b-kube-api-access-t4g96\") pod \"maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:32.201302 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.201204 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwq5v\" (UniqueName: \"kubernetes.io/projected/ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df-kube-api-access-xwq5v\") pod \"test-curl-8962-44aa17\" (UID: \"ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df\") " pod="opendatahub/test-curl-8962-44aa17" Oct 30 16:50:32.262712 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.262672 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:32.303074 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.303024 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwq5v\" (UniqueName: \"kubernetes.io/projected/ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df-kube-api-access-xwq5v\") pod \"test-curl-8962-44aa17\" (UID: \"ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df\") " pod="opendatahub/test-curl-8962-44aa17" Oct 30 16:50:32.328451 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.328417 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwq5v\" (UniqueName: \"kubernetes.io/projected/ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df-kube-api-access-xwq5v\") pod \"test-curl-8962-44aa17\" (UID: \"ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df\") " pod="opendatahub/test-curl-8962-44aa17" Oct 30 16:50:32.374841 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.374809 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-44aa17" Oct 30 16:50:32.584276 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:50:32.584243 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46fba546_bf94_4362_9b30_16d0aa688d8b.slice/crio-2cc270c9d1a3110661376705346c94f5d53b24c4e08905c94806c3aeaa6f8630 WatchSource:0}: Error finding container 2cc270c9d1a3110661376705346c94f5d53b24c4e08905c94806c3aeaa6f8630: Status 404 returned error can't find the container with id 2cc270c9d1a3110661376705346c94f5d53b24c4e08905c94806c3aeaa6f8630 Oct 30 16:50:32.584550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.584530 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z"] Oct 30 16:50:32.700716 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:50:32.700685 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae7bfa70_0f76_49c7_9e4f_a1fea6ad62df.slice/crio-ce4453ecd23bd9e0cc1585b5efa317f2ada0d151a1d9bf20a05e8b01cc8ca3b0 WatchSource:0}: Error finding container ce4453ecd23bd9e0cc1585b5efa317f2ada0d151a1d9bf20a05e8b01cc8ca3b0: Status 404 returned error can't find the container with id ce4453ecd23bd9e0cc1585b5efa317f2ada0d151a1d9bf20a05e8b01cc8ca3b0 Oct 30 16:50:32.701868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:32.701842 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8962-44aa17"] Oct 30 16:50:33.208493 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:33.208455 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" event={"ID":"46fba546-bf94-4362-9b30-16d0aa688d8b","Type":"ContainerStarted","Data":"a8aee8bc11f5c18e2018e7d9eb1458920be90b3595c41bd44635ea69bd3bef48"} Oct 30 16:50:33.208996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:33.208553 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" event={"ID":"46fba546-bf94-4362-9b30-16d0aa688d8b","Type":"ContainerStarted","Data":"2cc270c9d1a3110661376705346c94f5d53b24c4e08905c94806c3aeaa6f8630"} Oct 30 16:50:33.208996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:33.208589 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:33.209868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:33.209841 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8962-44aa17" event={"ID":"ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df","Type":"ContainerStarted","Data":"ce4453ecd23bd9e0cc1585b5efa317f2ada0d151a1d9bf20a05e8b01cc8ca3b0"} Oct 30 16:50:33.252415 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:33.252363 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" podStartSLOduration=1.783435271 podStartE2EDuration="2.252350476s" totalImagesPullingTime="468.915205ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:33.250284858 +0000 UTC m=+1742.400565859" watchObservedRunningTime="2026-10-30 16:50:33.252350476 +0000 UTC m=+1742.402631464" Oct 30 16:50:34.219972 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:34.219936 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8962-44aa17" event={"ID":"ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df","Type":"ContainerStarted","Data":"aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060"} Oct 30 16:50:34.258307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:34.258250 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8962-44aa17" podStartSLOduration=1.309278783 podStartE2EDuration="2.25823612s" totalImagesPullingTime="948.957337ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:34.256176378 +0000 UTC m=+1743.406457368" watchObservedRunningTime="2026-10-30 16:50:34.25823612 +0000 UTC m=+1743.408517107" Oct 30 16:50:35.458372 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:35.457924 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8962-44aa17"] Oct 30 16:50:35.470328 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:35.470256 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8962-44aa17"] Oct 30 16:50:36.288890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:36.288843 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8962-44aa17" podUID="ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df" containerName="test-curl-8962-44aa17" containerID="cri-o://aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060" gracePeriod=2 Oct 30 16:50:36.303766 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:50:36.303731 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8962-44aa17\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df" pod="opendatahub/test-curl-8962-44aa17" Oct 30 16:50:38.560658 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:38.560526 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-44aa17" Oct 30 16:50:38.655521 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:38.655488 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df-kube-api-access-xwq5v\" (UniqueName: \"kubernetes.io/projected/ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df-kube-api-access-xwq5v\") pod \"ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df\" (UID: \"ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df\") " Oct 30 16:50:38.657872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:38.657847 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df-kube-api-access-xwq5v" pod "ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df" (UID: "ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df"). InnerVolumeSpecName "kube-api-access-xwq5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:50:38.758309 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:38.758278 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwq5v\" (UniqueName: \"kubernetes.io/projected/ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df-kube-api-access-xwq5v\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:50:38.970167 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:38.970088 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df" path="/var/lib/kubelet/pods/ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df/volumes" Oct 30 16:50:39.225307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:39.225239 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:50:39.315115 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:39.315079 6400 generic.go:356] "Generic (PLEG): container finished" podID="ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df" containerID="aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060" exitCode=137 Oct 30 16:50:39.315374 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:39.315361 6400 scope.go:122] "RemoveContainer" containerID="aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060" Oct 30 16:50:39.315529 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:39.315517 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8962-44aa17" Oct 30 16:50:39.330199 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:39.330175 6400 scope.go:122] "RemoveContainer" containerID="aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060" Oct 30 16:50:39.330549 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:50:39.330521 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060\": container with ID starting with aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060 not found: ID does not exist" containerID="aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060" Oct 30 16:50:39.330743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:39.330717 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060"} err="failed to get container status \"aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060\": rpc error: code = NotFound desc = could not find container \"aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060\": container with ID starting with aee21cc829322f99562a04d92aab9da3fb4ef8d23b8744d8767db33cc1970060 not found: ID does not exist" Oct 30 16:50:46.778647 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.778589 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6"] Oct 30 16:50:46.780102 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.780076 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df" containerName="test-curl-8962-44aa17" Oct 30 16:50:46.780102 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.780102 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df" containerName="test-curl-8962-44aa17" Oct 30 16:50:46.780415 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.780396 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae7bfa70-0f76-49c7-9e4f-a1fea6ad62df" containerName="test-curl-8962-44aa17" Oct 30 16:50:46.786244 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.786216 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:46.820759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.820722 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6"] Oct 30 16:50:46.923497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.923180 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:46.923497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.923268 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blwbc\" (UniqueName: \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-kube-api-access-blwbc\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:46.923497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.923321 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500bd644-52a0-4fab-a8f7-c5286b16eb72-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:46.923497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.923357 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:46.923497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.923438 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:46.923497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.923466 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:46.923497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.923501 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:46.924065 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.923809 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:46.924065 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:46.923841 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.025279 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.025237 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500bd644-52a0-4fab-a8f7-c5286b16eb72-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.026531 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.026414 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500bd644-52a0-4fab-a8f7-c5286b16eb72-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.026761 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.026740 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.027060 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.027019 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.028514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.028492 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.028762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.028701 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.029105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.027478 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.029258 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.029190 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.029396 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.029379 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.030007 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.029982 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.030126 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.030056 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blwbc\" (UniqueName: \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-kube-api-access-blwbc\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.030193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.029785 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.030263 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.029467 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.030620 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.029494 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.056902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.054225 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.075649 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.075148 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.075828 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.075783 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.079622 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.079100 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blwbc\" (UniqueName: \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-kube-api-access-blwbc\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.109199 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.109163 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:47.454992 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:50:47.454958 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod500bd644_52a0_4fab_a8f7_c5286b16eb72.slice/crio-ae7f3f78718667f64d5808459e73bcadc6cec8cd392e24db82827016e30755d9 WatchSource:0}: Error finding container ae7f3f78718667f64d5808459e73bcadc6cec8cd392e24db82827016e30755d9: Status 404 returned error can't find the container with id ae7f3f78718667f64d5808459e73bcadc6cec8cd392e24db82827016e30755d9 Oct 30 16:50:47.455562 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.455469 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6"] Oct 30 16:50:47.457822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.457789 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:47.457919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.457851 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:47.457919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.457881 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:50:47.487665 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:47.487637 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" event={"ID":"500bd644-52a0-4fab-a8f7-c5286b16eb72","Type":"ContainerStarted","Data":"ae7f3f78718667f64d5808459e73bcadc6cec8cd392e24db82827016e30755d9"} Oct 30 16:50:48.498151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:48.498108 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" event={"ID":"500bd644-52a0-4fab-a8f7-c5286b16eb72","Type":"ContainerStarted","Data":"66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812"} Oct 30 16:50:48.561195 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:48.561140 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" podStartSLOduration=2.561125874 podStartE2EDuration="2.561125874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:50:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:50:48.56047944 +0000 UTC m=+1757.710760428" watchObservedRunningTime="2026-10-30 16:50:48.561125874 +0000 UTC m=+1757.711406861" Oct 30 16:50:49.115625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:49.113956 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:49.135627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:49.130776 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:50:49.135627 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:49.130859 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" podUID="500bd644-52a0-4fab-a8f7-c5286b16eb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:50:50.113626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:50.112991 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:50:50.113626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:50.113052 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" podUID="500bd644-52a0-4fab-a8f7-c5286b16eb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:50:51.110228 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:51.110188 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:50:51.110490 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:51.110459 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" podUID="500bd644-52a0-4fab-a8f7-c5286b16eb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:50:52.112631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:52.111017 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:50:52.112631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:52.111094 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" podUID="500bd644-52a0-4fab-a8f7-c5286b16eb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:50:53.128628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:53.121987 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body= Oct 30 16:50:53.128628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:53.122055 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" podUID="500bd644-52a0-4fab-a8f7-c5286b16eb72" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" Oct 30 16:50:54.349716 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:54.349681 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:54.350581 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:54.350556 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:54.351753 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:50:54.351723 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:50:54.603188 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:50:54.603099 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:51:01.785161 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.785123 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb"] Oct 30 16:51:01.789754 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.789731 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:01.821063 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.821029 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb"] Oct 30 16:51:01.940351 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.940320 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:01.940526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.940396 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:01.940526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.940436 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:01.940630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.940526 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:01.940630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.940573 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:01.940740 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.940647 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:01.940740 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.940697 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:01.940817 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.940766 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbkwm\" (UniqueName: \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-kube-api-access-lbkwm\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:01.940852 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:01.940837 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042127 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042039 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042127 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042077 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042127 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042098 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042138 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042175 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042315 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042394 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042440 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbkwm\" (UniqueName: \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-kube-api-access-lbkwm\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042487 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-certs\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042532 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042809 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042563 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.042974 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042943 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-credential-socket\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.043276 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.042999 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istiod-ca-cert\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.043276 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.043057 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-data\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.045102 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.045081 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-envoy\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.045102 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.045088 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-podinfo\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.065964 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.065942 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-token\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.066080 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.066020 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbkwm\" (UniqueName: \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-kube-api-access-lbkwm\") pod \"e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.105213 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.105181 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:02.437247 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:51:02.437218 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36ccb16_b0fc_4cec_a0cb_0bc8f50a8518.slice/crio-e70b37d915bfe89fb3bcfa7b10adf95535d9b699875f99e46f728f0775a4ff6b WatchSource:0}: Error finding container e70b37d915bfe89fb3bcfa7b10adf95535d9b699875f99e46f728f0775a4ff6b: Status 404 returned error can't find the container with id e70b37d915bfe89fb3bcfa7b10adf95535d9b699875f99e46f728f0775a4ff6b Oct 30 16:51:02.437944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.437916 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb"] Oct 30 16:51:02.440328 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.440291 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:51:02.440421 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.440361 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:51:02.440421 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.440390 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:51:02.667201 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.667162 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" event={"ID":"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518","Type":"ContainerStarted","Data":"255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84"} Oct 30 16:51:02.667413 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.667220 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" event={"ID":"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518","Type":"ContainerStarted","Data":"e70b37d915bfe89fb3bcfa7b10adf95535d9b699875f99e46f728f0775a4ff6b"} Oct 30 16:51:02.712660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:02.712503 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" podStartSLOduration=1.71248874 podStartE2EDuration="1.71248874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:02.711660532 +0000 UTC m=+1771.861941533" watchObservedRunningTime="2026-10-30 16:51:02.71248874 +0000 UTC m=+1771.862769727" Oct 30 16:51:03.105296 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:03.105255 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:03.107211 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:03.107183 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:51:03.107341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:03.107252 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:51:04.105768 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:04.105736 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:51:04.106237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:04.105817 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:51:05.105533 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:05.105496 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:51:05.105743 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:05.105557 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:51:06.106151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:06.106107 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:51:06.106686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:06.106170 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:51:07.106075 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:07.106036 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:51:07.106267 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:07.106094 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:51:08.106326 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:08.106290 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:51:08.106761 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:08.106352 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:51:09.105976 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:09.105934 6400 patch_prober.go:29] interesting pod/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" start-of-body= Oct 30 16:51:09.106193 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:09.105994 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.116:15021/healthz/ready\": dial tcp 10.134.0.116:15021: connect: connection refused" Oct 30 16:51:10.110883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:10.110850 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:10.111342 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:10.111289 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:10.111998 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:10.111974 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:17.274922 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.274890 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s"] Oct 30 16:51:17.280400 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.280380 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.314874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.314846 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-668047-serving-cert\"" Oct 30 16:51:17.319286 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.319259 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s"] Oct 30 16:51:17.419797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.419758 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb133fc0-29a7-4a23-b69c-66e10cacf293-openshift-service-ca\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.420017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.419811 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs7kp\" (UniqueName: \"kubernetes.io/projected/fb133fc0-29a7-4a23-b69c-66e10cacf293-kube-api-access-cs7kp\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.420017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.419960 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-maas-api-tls\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.420017 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.420007 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-metrics-certs\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.521666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.521580 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb133fc0-29a7-4a23-b69c-66e10cacf293-openshift-service-ca\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.521821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.521689 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs7kp\" (UniqueName: \"kubernetes.io/projected/fb133fc0-29a7-4a23-b69c-66e10cacf293-kube-api-access-cs7kp\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.521821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.521758 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-maas-api-tls\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.521821 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.521789 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-metrics-certs\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.521993 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:51:17.521896 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-serving-cert: secret "maas-api-e2e-inf-a-668047-serving-cert" not found Oct 30 16:51:17.521993 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:51:17.521970 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-maas-api-tls podName:fb133fc0-29a7-4a23-b69c-66e10cacf293 nodeName:}" failed. No retries permitted until 2026-10-30 16:51:18.021950573 +0000 UTC m=+1787.172231556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-maas-api-tls") pod "maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" (UID: "fb133fc0-29a7-4a23-b69c-66e10cacf293") : secret "maas-api-e2e-inf-a-668047-serving-cert" not found Oct 30 16:51:17.522258 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.522237 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb133fc0-29a7-4a23-b69c-66e10cacf293-openshift-service-ca\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.525114 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.525056 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-metrics-certs\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:17.547749 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:17.547716 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs7kp\" (UniqueName: \"kubernetes.io/projected/fb133fc0-29a7-4a23-b69c-66e10cacf293-kube-api-access-cs7kp\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:18.030904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:18.030869 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-maas-api-tls\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:18.033848 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:18.033817 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-maas-api-tls\") pod \"maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:18.196682 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:18.196637 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:18.551517 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:51:18.551485 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb133fc0_29a7_4a23_b69c_66e10cacf293.slice/crio-0ae3825bab236016d22f3496c2a59036f1233317f64456666d3145248804950a WatchSource:0}: Error finding container 0ae3825bab236016d22f3496c2a59036f1233317f64456666d3145248804950a: Status 404 returned error can't find the container with id 0ae3825bab236016d22f3496c2a59036f1233317f64456666d3145248804950a Oct 30 16:51:18.551975 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:18.551775 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s"] Oct 30 16:51:19.066280 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:19.066235 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" event={"ID":"fb133fc0-29a7-4a23-b69c-66e10cacf293","Type":"ContainerStarted","Data":"0ae3825bab236016d22f3496c2a59036f1233317f64456666d3145248804950a"} Oct 30 16:51:20.020722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:20.020681 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" event={"ID":"fb133fc0-29a7-4a23-b69c-66e10cacf293","Type":"ContainerStarted","Data":"78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff"} Oct 30 16:51:20.021254 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:20.020765 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:20.755133 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:20.755065 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" podStartSLOduration=3.206782453 podStartE2EDuration="3.755045406s" totalImagesPullingTime="548.262953ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:51:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:51:20.100948628 +0000 UTC m=+1789.251229617" watchObservedRunningTime="2026-10-30 16:51:20.755045406 +0000 UTC m=+1789.905326395" Oct 30 16:51:20.767721 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:20.761623 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn"] Oct 30 16:51:20.767721 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:20.761870 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" podUID="0cc052b6-68f0-41f0-8520-0b15744583dc" containerName="maas-api" containerID="cri-o://b0bdcccd80bb7ade326126a72645b6246fced1d6fe931d2fdaaefc92057b8dfc" gracePeriod=30 Oct 30 16:51:21.033019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.032925 6400 generic.go:356] "Generic (PLEG): container finished" podID="0cc052b6-68f0-41f0-8520-0b15744583dc" containerID="b0bdcccd80bb7ade326126a72645b6246fced1d6fe931d2fdaaefc92057b8dfc" exitCode=0 Oct 30 16:51:21.033019 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.032950 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" event={"ID":"0cc052b6-68f0-41f0-8520-0b15744583dc","Type":"ContainerDied","Data":"b0bdcccd80bb7ade326126a72645b6246fced1d6fe931d2fdaaefc92057b8dfc"} Oct 30 16:51:21.083231 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.083203 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:51:21.120675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.120645 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-metrics-certs\") pod \"0cc052b6-68f0-41f0-8520-0b15744583dc\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " Oct 30 16:51:21.121064 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.121040 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cc052b6-68f0-41f0-8520-0b15744583dc-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc052b6-68f0-41f0-8520-0b15744583dc-openshift-service-ca\") pod \"0cc052b6-68f0-41f0-8520-0b15744583dc\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " Oct 30 16:51:21.121470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.121449 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-maas-api-tls\") pod \"0cc052b6-68f0-41f0-8520-0b15744583dc\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " Oct 30 16:51:21.122264 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.122242 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cc052b6-68f0-41f0-8520-0b15744583dc-kube-api-access-jzqdp\" (UniqueName: \"kubernetes.io/projected/0cc052b6-68f0-41f0-8520-0b15744583dc-kube-api-access-jzqdp\") pod \"0cc052b6-68f0-41f0-8520-0b15744583dc\" (UID: \"0cc052b6-68f0-41f0-8520-0b15744583dc\") " Oct 30 16:51:21.122663 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.122515 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cc052b6-68f0-41f0-8520-0b15744583dc-openshift-service-ca" pod "0cc052b6-68f0-41f0-8520-0b15744583dc" (UID: "0cc052b6-68f0-41f0-8520-0b15744583dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:21.124177 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.123537 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cc052b6-68f0-41f0-8520-0b15744583dc-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:21.144888 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.144744 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-metrics-certs" pod "0cc052b6-68f0-41f0-8520-0b15744583dc" (UID: "0cc052b6-68f0-41f0-8520-0b15744583dc"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:21.144888 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.144847 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-maas-api-tls" pod "0cc052b6-68f0-41f0-8520-0b15744583dc" (UID: "0cc052b6-68f0-41f0-8520-0b15744583dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:51:21.147674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.147526 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cc052b6-68f0-41f0-8520-0b15744583dc-kube-api-access-jzqdp" pod "0cc052b6-68f0-41f0-8520-0b15744583dc" (UID: "0cc052b6-68f0-41f0-8520-0b15744583dc"). InnerVolumeSpecName "kube-api-access-jzqdp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:21.248632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.245732 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jzqdp\" (UniqueName: \"kubernetes.io/projected/0cc052b6-68f0-41f0-8520-0b15744583dc-kube-api-access-jzqdp\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:21.248632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.245772 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:21.248632 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:21.245787 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0cc052b6-68f0-41f0-8520-0b15744583dc-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:22.043343 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:22.043315 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" Oct 30 16:51:22.043951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:22.043343 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn" event={"ID":"0cc052b6-68f0-41f0-8520-0b15744583dc","Type":"ContainerDied","Data":"b0470c838dd7d0b76272f68f82880f7972591c5948fe0320c3f8d03b8ba90b75"} Oct 30 16:51:22.044095 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:22.043994 6400 scope.go:122] "RemoveContainer" containerID="b0bdcccd80bb7ade326126a72645b6246fced1d6fe931d2fdaaefc92057b8dfc" Oct 30 16:51:22.103471 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:22.103390 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn"] Oct 30 16:51:22.110870 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:22.110836 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-00fd1a-56d48c7ccb-9wsxn"] Oct 30 16:51:22.969385 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:22.969352 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cc052b6-68f0-41f0-8520-0b15744583dc" path="/var/lib/kubelet/pods/0cc052b6-68f0-41f0-8520-0b15744583dc/volumes" Oct 30 16:51:24.601322 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:51:24.601283 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:51:26.038958 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:26.038929 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:51:46.441354 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:46.441142 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr"] Oct 30 16:51:46.442992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:46.442956 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" containerID="cri-o://04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759" gracePeriod=30 Oct 30 16:51:46.457541 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:46.457057 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb"] Oct 30 16:51:46.457541 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:46.457299 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" containerID="cri-o://255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84" gracePeriod=30 Oct 30 16:51:46.467950 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:46.467766 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6"] Oct 30 16:51:46.468362 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:46.468329 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" podUID="500bd644-52a0-4fab-a8f7-c5286b16eb72" containerName="istio-proxy" containerID="cri-o://66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812" gracePeriod=30 Oct 30 16:51:51.787556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:51.770943 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj"] Oct 30 16:51:51.787556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:51.777780 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerName="istio-proxy" containerID="cri-o://646e311d360a39208f35e7bb38354e7c68b7bc201a96a0a7466be816d50ea8b3" gracePeriod=30 Oct 30 16:51:51.787556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:51.782528 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr"] Oct 30 16:51:51.787556 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:51.782767 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" containerID="cri-o://cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1" gracePeriod=30 Oct 30 16:51:51.834797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:51.834768 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:51:51.900779 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:51.900751 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:51.914533 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:51.914504 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:51:52.008483 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.008450 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-credential-socket\") pod \"f1899f96-b860-4c71-a10b-c36ee2587ace\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " Oct 30 16:51:52.008706 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.008501 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-certs\") pod \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " Oct 30 16:51:52.008706 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.008666 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-token\" (UniqueName: \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-token\") pod \"f1899f96-b860-4c71-a10b-c36ee2587ace\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " Oct 30 16:51:52.008841 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.008738 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-podinfo\") pod \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " Oct 30 16:51:52.008841 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.008801 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-kube-api-access-twgvp\" (UniqueName: \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-kube-api-access-twgvp\") pod \"f1899f96-b860-4c71-a10b-c36ee2587ace\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " Oct 30 16:51:52.008841 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.008823 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-credential-socket" pod "f1899f96-b860-4c71-a10b-c36ee2587ace" (UID: "f1899f96-b860-4c71-a10b-c36ee2587ace"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.009021 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.008859 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1899f96-b860-4c71-a10b-c36ee2587ace-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1899f96-b860-4c71-a10b-c36ee2587ace-istiod-ca-cert\") pod \"f1899f96-b860-4c71-a10b-c36ee2587ace\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " Oct 30 16:51:52.009021 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.008929 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-kube-api-access-lbkwm\" (UniqueName: \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-kube-api-access-lbkwm\") pod \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " Oct 30 16:51:52.009021 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.008993 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-socket\") pod \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " Oct 30 16:51:52.009194 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.009065 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-credential-socket\") pod \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " Oct 30 16:51:52.009194 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.009100 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-token\" (UniqueName: \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-token\") pod \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " Oct 30 16:51:52.009311 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.009235 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-certs\") pod \"f1899f96-b860-4c71-a10b-c36ee2587ace\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " Oct 30 16:51:52.009311 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.009262 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-envoy\") pod \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " Oct 30 16:51:52.009655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.009627 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-certs" pod "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" (UID: "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.010696 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.010669 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-certs" pod "f1899f96-b860-4c71-a10b-c36ee2587ace" (UID: "f1899f96-b860-4c71-a10b-c36ee2587ace"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.013139 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.011838 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-credential-socket" pod "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" (UID: "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.013483 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.013459 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1899f96-b860-4c71-a10b-c36ee2587ace-istiod-ca-cert" pod "f1899f96-b860-4c71-a10b-c36ee2587ace" (UID: "f1899f96-b860-4c71-a10b-c36ee2587ace"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:52.013958 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.013920 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-socket" pod "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" (UID: "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.013958 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.010406 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istiod-ca-cert" pod "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" (UID: "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:52.013958 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.009961 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istiod-ca-cert\") pod \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " Oct 30 16:51:52.014163 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.014048 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-envoy\") pod \"f1899f96-b860-4c71-a10b-c36ee2587ace\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " Oct 30 16:51:52.014163 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.014086 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-data\") pod \"f1899f96-b860-4c71-a10b-c36ee2587ace\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " Oct 30 16:51:52.014163 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.014139 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-podinfo\") pod \"f1899f96-b860-4c71-a10b-c36ee2587ace\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " Oct 30 16:51:52.014326 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.014174 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-data\") pod \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\" (UID: \"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518\") " Oct 30 16:51:52.014326 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.014225 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-podinfo" pod "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" (UID: "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:52.014326 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.014229 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-socket\") pod \"f1899f96-b860-4c71-a10b-c36ee2587ace\" (UID: \"f1899f96-b860-4c71-a10b-c36ee2587ace\") " Oct 30 16:51:52.014493 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.014422 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-socket" pod "f1899f96-b860-4c71-a10b-c36ee2587ace" (UID: "f1899f96-b860-4c71-a10b-c36ee2587ace"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.015833 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.015811 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.017003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.016849 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.017003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.016883 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.017003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.016899 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.017003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.016914 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.017003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.016928 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.017003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.016942 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.017003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.016956 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.017003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.016969 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f1899f96-b860-4c71-a10b-c36ee2587ace-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.017003 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.015826 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-data" pod "f1899f96-b860-4c71-a10b-c36ee2587ace" (UID: "f1899f96-b860-4c71-a10b-c36ee2587ace"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.017560 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.017129 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-data" pod "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" (UID: "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.022334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.022286 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-envoy" pod "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" (UID: "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.022674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.022569 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-token" pod "f1899f96-b860-4c71-a10b-c36ee2587ace" (UID: "f1899f96-b860-4c71-a10b-c36ee2587ace"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.023128 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.023103 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-podinfo" pod "f1899f96-b860-4c71-a10b-c36ee2587ace" (UID: "f1899f96-b860-4c71-a10b-c36ee2587ace"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:52.023385 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.023350 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-kube-api-access-lbkwm" pod "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" (UID: "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518"). InnerVolumeSpecName "kube-api-access-lbkwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.025423 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.025383 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-kube-api-access-twgvp" pod "f1899f96-b860-4c71-a10b-c36ee2587ace" (UID: "f1899f96-b860-4c71-a10b-c36ee2587ace"). InnerVolumeSpecName "kube-api-access-twgvp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.025833 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.025797 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-token" pod "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" (UID: "f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.026310 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.026283 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-envoy" pod "f1899f96-b860-4c71-a10b-c36ee2587ace" (UID: "f1899f96-b860-4c71-a10b-c36ee2587ace"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.214686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.129481 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-data\") pod \"500bd644-52a0-4fab-a8f7-c5286b16eb72\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " Oct 30 16:51:52.214686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.129550 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-kube-api-access-blwbc\" (UniqueName: \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-kube-api-access-blwbc\") pod \"500bd644-52a0-4fab-a8f7-c5286b16eb72\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " Oct 30 16:51:52.214686 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.139271 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-data" pod "500bd644-52a0-4fab-a8f7-c5286b16eb72" (UID: "500bd644-52a0-4fab-a8f7-c5286b16eb72"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.279714 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.244741 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-podinfo" pod "500bd644-52a0-4fab-a8f7-c5286b16eb72" (UID: "500bd644-52a0-4fab-a8f7-c5286b16eb72"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:52.279714 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.251284 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-kube-api-access-blwbc" pod "500bd644-52a0-4fab-a8f7-c5286b16eb72" (UID: "500bd644-52a0-4fab-a8f7-c5286b16eb72"). InnerVolumeSpecName "kube-api-access-blwbc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.284985 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-podinfo\") pod \"500bd644-52a0-4fab-a8f7-c5286b16eb72\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.285086 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-envoy\") pod \"500bd644-52a0-4fab-a8f7-c5286b16eb72\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.285208 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-credential-socket\") pod \"500bd644-52a0-4fab-a8f7-c5286b16eb72\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.285302 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-socket\") pod \"500bd644-52a0-4fab-a8f7-c5286b16eb72\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.285436 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/500bd644-52a0-4fab-a8f7-c5286b16eb72-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500bd644-52a0-4fab-a8f7-c5286b16eb72-istiod-ca-cert\") pod \"500bd644-52a0-4fab-a8f7-c5286b16eb72\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.285562 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-token\" (UniqueName: \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-token\") pod \"500bd644-52a0-4fab-a8f7-c5286b16eb72\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.285637 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-certs\") pod \"500bd644-52a0-4fab-a8f7-c5286b16eb72\" (UID: \"500bd644-52a0-4fab-a8f7-c5286b16eb72\") " Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.286977 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twgvp\" (UniqueName: \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-kube-api-access-twgvp\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287005 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lbkwm\" (UniqueName: \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-kube-api-access-lbkwm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287527 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287557 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287572 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287588 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287623 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287643 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287659 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287673 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blwbc\" (UniqueName: \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-kube-api-access-blwbc\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287688 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287710 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f1899f96-b860-4c71-a10b-c36ee2587ace-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.287904 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-socket" pod "500bd644-52a0-4fab-a8f7-c5286b16eb72" (UID: "500bd644-52a0-4fab-a8f7-c5286b16eb72"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.288239 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-certs" pod "500bd644-52a0-4fab-a8f7-c5286b16eb72" (UID: "500bd644-52a0-4fab-a8f7-c5286b16eb72"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.290682 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/500bd644-52a0-4fab-a8f7-c5286b16eb72-istiod-ca-cert" pod "500bd644-52a0-4fab-a8f7-c5286b16eb72" (UID: "500bd644-52a0-4fab-a8f7-c5286b16eb72"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:52.296625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.290965 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-credential-socket" pod "500bd644-52a0-4fab-a8f7-c5286b16eb72" (UID: "500bd644-52a0-4fab-a8f7-c5286b16eb72"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.309676 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.304451 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-token" pod "500bd644-52a0-4fab-a8f7-c5286b16eb72" (UID: "500bd644-52a0-4fab-a8f7-c5286b16eb72"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:52.309676 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.304933 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-envoy" pod "500bd644-52a0-4fab-a8f7-c5286b16eb72" (UID: "500bd644-52a0-4fab-a8f7-c5286b16eb72"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:52.388811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.388773 6400 generic.go:356] "Generic (PLEG): container finished" podID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerID="255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84" exitCode=0 Oct 30 16:51:52.389013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.388836 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" Oct 30 16:51:52.389013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.388858 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" event={"ID":"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518","Type":"ContainerDied","Data":"255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84"} Oct 30 16:51:52.389013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.388955 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.389013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.388974 6400 scope.go:122] "RemoveContainer" containerID="255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84" Oct 30 16:51:52.389232 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.388978 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.389232 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.388957 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb" event={"ID":"f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518","Type":"ContainerDied","Data":"e70b37d915bfe89fb3bcfa7b10adf95535d9b699875f99e46f728f0775a4ff6b"} Oct 30 16:51:52.389232 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.389146 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.389520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.389492 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.389652 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.389524 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/500bd644-52a0-4fab-a8f7-c5286b16eb72-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.389652 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.389542 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/500bd644-52a0-4fab-a8f7-c5286b16eb72-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:52.390514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.390485 6400 generic.go:356] "Generic (PLEG): container finished" podID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerID="04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759" exitCode=0 Oct 30 16:51:52.390668 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.390553 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" Oct 30 16:51:52.390668 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.390564 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" event={"ID":"f1899f96-b860-4c71-a10b-c36ee2587ace","Type":"ContainerDied","Data":"04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759"} Oct 30 16:51:52.390668 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.390637 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr" event={"ID":"f1899f96-b860-4c71-a10b-c36ee2587ace","Type":"ContainerDied","Data":"07bbe4a297a9de4efe21c9dcd1f4741b45a9650dad3e2c046bfdb84b6c7b9e60"} Oct 30 16:51:52.392281 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.392247 6400 generic.go:356] "Generic (PLEG): container finished" podID="500bd644-52a0-4fab-a8f7-c5286b16eb72" containerID="66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812" exitCode=0 Oct 30 16:51:52.392390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.392327 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" Oct 30 16:51:52.392390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.392335 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" event={"ID":"500bd644-52a0-4fab-a8f7-c5286b16eb72","Type":"ContainerDied","Data":"66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812"} Oct 30 16:51:52.392390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.392379 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6" event={"ID":"500bd644-52a0-4fab-a8f7-c5286b16eb72","Type":"ContainerDied","Data":"ae7f3f78718667f64d5808459e73bcadc6cec8cd392e24db82827016e30755d9"} Oct 30 16:51:52.401788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.401763 6400 scope.go:122] "RemoveContainer" containerID="255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84" Oct 30 16:51:52.402133 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:51:52.402112 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84\": container with ID starting with 255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84 not found: ID does not exist" containerID="255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84" Oct 30 16:51:52.402197 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.402144 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84"} err="failed to get container status \"255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84\": rpc error: code = NotFound desc = could not find container \"255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84\": container with ID starting with 255db4e263d3b42dc1a98ad32ef686eb922a65e41147c9cf4bb159f9844b0d84 not found: ID does not exist" Oct 30 16:51:52.402197 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.402166 6400 scope.go:122] "RemoveContainer" containerID="04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759" Oct 30 16:51:52.413342 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.413324 6400 scope.go:122] "RemoveContainer" containerID="04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759" Oct 30 16:51:52.413705 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:51:52.413685 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759\": container with ID starting with 04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759 not found: ID does not exist" containerID="04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759" Oct 30 16:51:52.413763 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.413716 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759"} err="failed to get container status \"04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759\": rpc error: code = NotFound desc = could not find container \"04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759\": container with ID starting with 04fff39b76b4c4ac0b117e13a347aafe9c591ef9e5160fe5d92f788fee09e759 not found: ID does not exist" Oct 30 16:51:52.413763 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.413734 6400 scope.go:122] "RemoveContainer" containerID="66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812" Oct 30 16:51:52.424630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.424590 6400 scope.go:122] "RemoveContainer" containerID="66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812" Oct 30 16:51:52.425016 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:51:52.424995 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812\": container with ID starting with 66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812 not found: ID does not exist" containerID="66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812" Oct 30 16:51:52.425079 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.425024 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812"} err="failed to get container status \"66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812\": rpc error: code = NotFound desc = could not find container \"66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812\": container with ID starting with 66bd2364d2a0e9da749b1fab8fc6a5ea7ce3f922bca86f989bee99116d7e5812 not found: ID does not exist" Oct 30 16:51:52.470968 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.470884 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6"] Oct 30 16:51:52.483049 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.483017 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-85tl6"] Oct 30 16:51:52.511282 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.511255 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb"] Oct 30 16:51:52.514983 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.514959 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-m9gjb"] Oct 30 16:51:52.548809 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.548775 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr"] Oct 30 16:51:52.552365 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.552344 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b8af660d-openshift-default-69fbc6b79c-d9khr"] Oct 30 16:51:52.972891 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.972850 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="500bd644-52a0-4fab-a8f7-c5286b16eb72" path="/var/lib/kubelet/pods/500bd644-52a0-4fab-a8f7-c5286b16eb72/volumes" Oct 30 16:51:52.973473 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.973458 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" path="/var/lib/kubelet/pods/f1899f96-b860-4c71-a10b-c36ee2587ace/volumes" Oct 30 16:51:52.973921 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:52.973908 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" path="/var/lib/kubelet/pods/f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518/volumes" Oct 30 16:51:54.607474 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:51:54.607443 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:51:57.063822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.063746 6400 patch_prober.go:29] interesting pod/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" start-of-body= Oct 30 16:51:57.063822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.063804 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.60:15021/healthz/ready\": dial tcp 10.134.0.60:15021: connect: connection refused" Oct 30 16:51:57.068797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.068773 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:51:57.180051 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180018 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4256fb11-e481-46fb-9964-1706f25e8e71-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4256fb11-e481-46fb-9964-1706f25e8e71-istio-podinfo\") pod \"4256fb11-e481-46fb-9964-1706f25e8e71\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " Oct 30 16:51:57.180255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180116 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-data\") pod \"4256fb11-e481-46fb-9964-1706f25e8e71\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " Oct 30 16:51:57.180255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180175 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-istio-token\" (UniqueName: \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-istio-token\") pod \"4256fb11-e481-46fb-9964-1706f25e8e71\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " Oct 30 16:51:57.180255 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180226 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-kube-api-access-5spqn\" (UniqueName: \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-kube-api-access-5spqn\") pod \"4256fb11-e481-46fb-9964-1706f25e8e71\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " Oct 30 16:51:57.180434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180314 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-credential-socket\") pod \"4256fb11-e481-46fb-9964-1706f25e8e71\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " Oct 30 16:51:57.180434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180373 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-certs\") pod \"4256fb11-e481-46fb-9964-1706f25e8e71\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " Oct 30 16:51:57.180434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180406 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-socket\") pod \"4256fb11-e481-46fb-9964-1706f25e8e71\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " Oct 30 16:51:57.180585 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180473 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-data" pod "4256fb11-e481-46fb-9964-1706f25e8e71" (UID: "4256fb11-e481-46fb-9964-1706f25e8e71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:57.180585 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180483 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4256fb11-e481-46fb-9964-1706f25e8e71-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4256fb11-e481-46fb-9964-1706f25e8e71-istiod-ca-cert\") pod \"4256fb11-e481-46fb-9964-1706f25e8e71\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " Oct 30 16:51:57.180585 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180545 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-envoy\") pod \"4256fb11-e481-46fb-9964-1706f25e8e71\" (UID: \"4256fb11-e481-46fb-9964-1706f25e8e71\") " Oct 30 16:51:57.180760 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180708 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-credential-socket" pod "4256fb11-e481-46fb-9964-1706f25e8e71" (UID: "4256fb11-e481-46fb-9964-1706f25e8e71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:57.181152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.180810 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4256fb11-e481-46fb-9964-1706f25e8e71-istiod-ca-cert" pod "4256fb11-e481-46fb-9964-1706f25e8e71" (UID: "4256fb11-e481-46fb-9964-1706f25e8e71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:57.181152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.181046 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-certs" pod "4256fb11-e481-46fb-9964-1706f25e8e71" (UID: "4256fb11-e481-46fb-9964-1706f25e8e71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:57.181152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.181107 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-socket" pod "4256fb11-e481-46fb-9964-1706f25e8e71" (UID: "4256fb11-e481-46fb-9964-1706f25e8e71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:57.181514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.181490 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.181789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.181522 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.181789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.181535 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.181789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.181548 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.181789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.181560 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4256fb11-e481-46fb-9964-1706f25e8e71-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.183660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.183588 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-envoy" pod "4256fb11-e481-46fb-9964-1706f25e8e71" (UID: "4256fb11-e481-46fb-9964-1706f25e8e71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:57.183825 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.183664 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-istio-token" pod "4256fb11-e481-46fb-9964-1706f25e8e71" (UID: "4256fb11-e481-46fb-9964-1706f25e8e71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:57.183825 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.183815 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-kube-api-access-5spqn" pod "4256fb11-e481-46fb-9964-1706f25e8e71" (UID: "4256fb11-e481-46fb-9964-1706f25e8e71"). InnerVolumeSpecName "kube-api-access-5spqn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:57.184064 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.184047 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4256fb11-e481-46fb-9964-1706f25e8e71-istio-podinfo" pod "4256fb11-e481-46fb-9964-1706f25e8e71" (UID: "4256fb11-e481-46fb-9964-1706f25e8e71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:57.282850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.282814 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4256fb11-e481-46fb-9964-1706f25e8e71-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.282850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.282853 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4256fb11-e481-46fb-9964-1706f25e8e71-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.283049 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.282866 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.283049 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.282876 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5spqn\" (UniqueName: \"kubernetes.io/projected/4256fb11-e481-46fb-9964-1706f25e8e71-kube-api-access-5spqn\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.436554 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.436519 6400 generic.go:356] "Generic (PLEG): container finished" podID="4256fb11-e481-46fb-9964-1706f25e8e71" containerID="cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1" exitCode=0 Oct 30 16:51:57.436748 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.436568 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" Oct 30 16:51:57.436748 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.436596 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" event={"ID":"4256fb11-e481-46fb-9964-1706f25e8e71","Type":"ContainerDied","Data":"cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1"} Oct 30 16:51:57.436748 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.436678 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr" event={"ID":"4256fb11-e481-46fb-9964-1706f25e8e71","Type":"ContainerDied","Data":"4d5791244c0097b1abe7b6095dce0fe1855f9ba5425d90a2ca15b518c3a8580e"} Oct 30 16:51:57.436748 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.436709 6400 scope.go:122] "RemoveContainer" containerID="cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1" Oct 30 16:51:57.437871 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.437760 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:51:57.438441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.438416 6400 generic.go:356] "Generic (PLEG): container finished" podID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerID="646e311d360a39208f35e7bb38354e7c68b7bc201a96a0a7466be816d50ea8b3" exitCode=0 Oct 30 16:51:57.438542 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.438478 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" event={"ID":"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4","Type":"ContainerDied","Data":"646e311d360a39208f35e7bb38354e7c68b7bc201a96a0a7466be816d50ea8b3"} Oct 30 16:51:57.447983 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.447965 6400 scope.go:122] "RemoveContainer" containerID="cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1" Oct 30 16:51:57.448284 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:51:57.448265 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1\": container with ID starting with cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1 not found: ID does not exist" containerID="cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1" Oct 30 16:51:57.448328 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.448293 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1"} err="failed to get container status \"cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1\": rpc error: code = NotFound desc = could not find container \"cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1\": container with ID starting with cd11e440f0fd54d4de40e730926dbae6e5b4efe3f263dd8928fc2b93b36ddfa1 not found: ID does not exist" Oct 30 16:51:57.448328 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.448310 6400 scope.go:122] "RemoveContainer" containerID="646e311d360a39208f35e7bb38354e7c68b7bc201a96a0a7466be816d50ea8b3" Oct 30 16:51:57.573457 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.573423 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr"] Oct 30 16:51:57.584893 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.584771 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-wlwhr"] Oct 30 16:51:57.588529 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.588505 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-envoy\") pod \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " Oct 30 16:51:57.588708 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.588555 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-podinfo\") pod \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " Oct 30 16:51:57.588708 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.588586 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istiod-ca-cert\") pod \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " Oct 30 16:51:57.588708 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.588647 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-certs\") pod \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " Oct 30 16:51:57.588880 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.588822 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-token\" (UniqueName: \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-token\") pod \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " Oct 30 16:51:57.588937 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.588882 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-socket\") pod \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " Oct 30 16:51:57.589012 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.588995 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-credential-socket\") pod \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " Oct 30 16:51:57.589066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589043 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-kube-api-access-5bt68\" (UniqueName: \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-kube-api-access-5bt68\") pod \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " Oct 30 16:51:57.589123 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589062 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istiod-ca-cert" pod "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" (UID: "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:51:57.589123 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589085 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-data\") pod \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\" (UID: \"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4\") " Oct 30 16:51:57.589123 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589088 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-certs" pod "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" (UID: "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:57.589368 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589324 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-socket" pod "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" (UID: "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:57.589470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589433 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-credential-socket" pod "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" (UID: "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:57.589779 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589760 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.589881 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589785 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.589881 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589807 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.589881 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589819 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.590039 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.589933 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-data" pod "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" (UID: "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:57.591334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.591304 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-podinfo" pod "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" (UID: "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:51:57.591440 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.591374 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-token" pod "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" (UID: "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:57.591776 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.591744 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-envoy" pod "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" (UID: "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:51:57.592191 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.592176 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-kube-api-access-5bt68" pod "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" (UID: "4350912f-cfe4-4ca9-8e9e-43dc41b25cf4"). InnerVolumeSpecName "kube-api-access-5bt68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:51:57.691992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.691774 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bt68\" (UniqueName: \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-kube-api-access-5bt68\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.691992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.691824 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.691992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.691852 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.691992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.691867 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:57.691992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:57.691880 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:51:58.449214 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:58.449174 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" event={"ID":"4350912f-cfe4-4ca9-8e9e-43dc41b25cf4","Type":"ContainerDied","Data":"0c69b5261872597e09ba42b298817daa6d289f44c9e891067dcde2635e21c1b4"} Oct 30 16:51:58.449631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:58.449292 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj" Oct 30 16:51:58.529895 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:58.529863 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj"] Oct 30 16:51:58.540819 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:58.540793 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-h8dqj"] Oct 30 16:51:58.969562 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:58.969524 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" path="/var/lib/kubelet/pods/4256fb11-e481-46fb-9964-1706f25e8e71/volumes" Oct 30 16:51:58.970486 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:51:58.970461 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" path="/var/lib/kubelet/pods/4350912f-cfe4-4ca9-8e9e-43dc41b25cf4/volumes" Oct 30 16:52:00.484701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.484668 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7"] Oct 30 16:52:00.485485 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485463 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerName="istio-proxy" Oct 30 16:52:00.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485487 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerName="istio-proxy" Oct 30 16:52:00.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485534 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" Oct 30 16:52:00.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485542 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" Oct 30 16:52:00.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485562 6400 cpu_manager.go:414] "containerMap: removing container" podUID="500bd644-52a0-4fab-a8f7-c5286b16eb72" containerName="istio-proxy" Oct 30 16:52:00.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485570 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="500bd644-52a0-4fab-a8f7-c5286b16eb72" containerName="istio-proxy" Oct 30 16:52:00.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485584 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" Oct 30 16:52:00.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485591 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" Oct 30 16:52:00.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485621 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0cc052b6-68f0-41f0-8520-0b15744583dc" containerName="maas-api" Oct 30 16:52:00.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485628 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cc052b6-68f0-41f0-8520-0b15744583dc" containerName="maas-api" Oct 30 16:52:00.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485652 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" Oct 30 16:52:00.486156 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485659 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" Oct 30 16:52:00.486156 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485843 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1899f96-b860-4c71-a10b-c36ee2587ace" containerName="istio-proxy" Oct 30 16:52:00.486156 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485858 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="500bd644-52a0-4fab-a8f7-c5286b16eb72" containerName="istio-proxy" Oct 30 16:52:00.486156 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485870 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cc052b6-68f0-41f0-8520-0b15744583dc" containerName="maas-api" Oct 30 16:52:00.486156 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485881 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4256fb11-e481-46fb-9964-1706f25e8e71" containerName="istio-proxy" Oct 30 16:52:00.486156 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485895 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4350912f-cfe4-4ca9-8e9e-43dc41b25cf4" containerName="istio-proxy" Oct 30 16:52:00.486156 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.485911 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f36ccb16-b0fc-4cec-a0cb-0bc8f50a8518" containerName="istio-proxy" Oct 30 16:52:00.492329 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.492303 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.518238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.518205 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-480a60-serving-cert\"" Oct 30 16:52:00.534641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.534576 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7"] Oct 30 16:52:00.593654 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.593569 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-maas-api-tls\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.593841 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.593700 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-metrics-certs\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.593841 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.593826 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jsq9\" (UniqueName: \"kubernetes.io/projected/e6b96ac8-4863-48fc-80bc-d93b140a3095-kube-api-access-7jsq9\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.593982 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.593933 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b96ac8-4863-48fc-80bc-d93b140a3095-openshift-service-ca\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.715628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.710660 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jsq9\" (UniqueName: \"kubernetes.io/projected/e6b96ac8-4863-48fc-80bc-d93b140a3095-kube-api-access-7jsq9\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.715628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.710792 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b96ac8-4863-48fc-80bc-d93b140a3095-openshift-service-ca\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.715628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.710960 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-maas-api-tls\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.715628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.711044 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-metrics-certs\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.715628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.712533 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b96ac8-4863-48fc-80bc-d93b140a3095-openshift-service-ca\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.735674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.727450 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-maas-api-tls\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.765630 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.761013 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-metrics-certs\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.780652 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.771391 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jsq9\" (UniqueName: \"kubernetes.io/projected/e6b96ac8-4863-48fc-80bc-d93b140a3095-kube-api-access-7jsq9\") pod \"maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:00.827870 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:00.826286 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:01.189732 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:52:01.188623 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b96ac8_4863_48fc_80bc_d93b140a3095.slice/crio-9be378c785fa9f9c4c1086de18338c33f2b0538d64a2ee1b73afaa35d11602dd WatchSource:0}: Error finding container 9be378c785fa9f9c4c1086de18338c33f2b0538d64a2ee1b73afaa35d11602dd: Status 404 returned error can't find the container with id 9be378c785fa9f9c4c1086de18338c33f2b0538d64a2ee1b73afaa35d11602dd Oct 30 16:52:01.189994 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:01.189924 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7"] Oct 30 16:52:01.530359 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:01.530325 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" event={"ID":"e6b96ac8-4863-48fc-80bc-d93b140a3095","Type":"ContainerStarted","Data":"9be378c785fa9f9c4c1086de18338c33f2b0538d64a2ee1b73afaa35d11602dd"} Oct 30 16:52:02.594408 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:02.594363 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" event={"ID":"e6b96ac8-4863-48fc-80bc-d93b140a3095","Type":"ContainerStarted","Data":"d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd"} Oct 30 16:52:02.596224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:02.596201 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:02.652932 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:02.652869 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" podStartSLOduration=2.299778146 podStartE2EDuration="2.652854605s" totalImagesPullingTime="353.076459ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:02.650585746 +0000 UTC m=+1831.800866735" watchObservedRunningTime="2026-10-30 16:52:02.652854605 +0000 UTC m=+1831.803135593" Oct 30 16:52:05.943636 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:05.943190 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b"] Oct 30 16:52:05.944154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:05.943885 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" containerName="istio-proxy" containerID="cri-o://51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c" gracePeriod=30 Oct 30 16:52:05.961114 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:05.955528 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh"] Oct 30 16:52:05.961114 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:05.955876 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" containerID="cri-o://f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df" gracePeriod=30 Oct 30 16:52:06.431108 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:06.431076 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4"] Oct 30 16:52:06.431359 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:06.431325 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" containerID="cri-o://8a8f273f36689061ad36b51c4b8fad050c198f1dce2d3b539897442af74236a4" gracePeriod=30 Oct 30 16:52:08.611569 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:08.611539 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:52:10.859701 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:10.859657 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:52:10.860177 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:10.859767 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:52:11.257955 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.257925 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:52:11.308849 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.308824 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:52:11.373703 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.373668 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-envoy\") pod \"a331a4b1-0c84-4850-a289-beda7882adbf\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " Oct 30 16:52:11.373880 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.373765 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-envoy\") pod \"6fcac11b-bb6f-4004-8830-e52f952f08a5\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " Oct 30 16:52:11.373880 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.373818 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-socket\") pod \"6fcac11b-bb6f-4004-8830-e52f952f08a5\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " Oct 30 16:52:11.373880 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.373851 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-socket\") pod \"a331a4b1-0c84-4850-a289-beda7882adbf\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " Oct 30 16:52:11.373992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.373881 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fcac11b-bb6f-4004-8830-e52f952f08a5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcac11b-bb6f-4004-8830-e52f952f08a5-istiod-ca-cert\") pod \"6fcac11b-bb6f-4004-8830-e52f952f08a5\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " Oct 30 16:52:11.373992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.373919 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-data\") pod \"6fcac11b-bb6f-4004-8830-e52f952f08a5\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " Oct 30 16:52:11.373992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.373952 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-kube-api-access-6pp79\" (UniqueName: \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-kube-api-access-6pp79\") pod \"a331a4b1-0c84-4850-a289-beda7882adbf\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " Oct 30 16:52:11.374109 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374007 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-certs\") pod \"a331a4b1-0c84-4850-a289-beda7882adbf\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " Oct 30 16:52:11.374109 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374078 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a331a4b1-0c84-4850-a289-beda7882adbf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a331a4b1-0c84-4850-a289-beda7882adbf-istiod-ca-cert\") pod \"a331a4b1-0c84-4850-a289-beda7882adbf\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " Oct 30 16:52:11.374214 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374118 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-kube-api-access-mr8tx\" (UniqueName: \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-kube-api-access-mr8tx\") pod \"6fcac11b-bb6f-4004-8830-e52f952f08a5\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " Oct 30 16:52:11.374214 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374196 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-credential-socket\") pod \"a331a4b1-0c84-4850-a289-beda7882adbf\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " Oct 30 16:52:11.374331 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374257 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-data\") pod \"a331a4b1-0c84-4850-a289-beda7882adbf\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " Oct 30 16:52:11.374331 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374310 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-data" pod "6fcac11b-bb6f-4004-8830-e52f952f08a5" (UID: "6fcac11b-bb6f-4004-8830-e52f952f08a5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.374434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374322 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-credential-socket\") pod \"6fcac11b-bb6f-4004-8830-e52f952f08a5\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " Oct 30 16:52:11.374538 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374510 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-socket" pod "6fcac11b-bb6f-4004-8830-e52f952f08a5" (UID: "6fcac11b-bb6f-4004-8830-e52f952f08a5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.374776 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374563 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-credential-socket" pod "6fcac11b-bb6f-4004-8830-e52f952f08a5" (UID: "6fcac11b-bb6f-4004-8830-e52f952f08a5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.375009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374983 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-certs" pod "a331a4b1-0c84-4850-a289-beda7882adbf" (UID: "a331a4b1-0c84-4850-a289-beda7882adbf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.375107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374778 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-socket" pod "a331a4b1-0c84-4850-a289-beda7882adbf" (UID: "a331a4b1-0c84-4850-a289-beda7882adbf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.375254 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.375135 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a331a4b1-0c84-4850-a289-beda7882adbf-istiod-ca-cert" pod "a331a4b1-0c84-4850-a289-beda7882adbf" (UID: "a331a4b1-0c84-4850-a289-beda7882adbf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:11.375322 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.374784 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-istio-token\" (UniqueName: \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-istio-token\") pod \"a331a4b1-0c84-4850-a289-beda7882adbf\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " Oct 30 16:52:11.375388 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.375355 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-data" pod "a331a4b1-0c84-4850-a289-beda7882adbf" (UID: "a331a4b1-0c84-4850-a289-beda7882adbf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.375580 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.375390 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-certs\") pod \"6fcac11b-bb6f-4004-8830-e52f952f08a5\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " Oct 30 16:52:11.375580 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.375481 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-token\" (UniqueName: \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-token\") pod \"6fcac11b-bb6f-4004-8830-e52f952f08a5\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " Oct 30 16:52:11.375580 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.375515 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-podinfo\") pod \"6fcac11b-bb6f-4004-8830-e52f952f08a5\" (UID: \"6fcac11b-bb6f-4004-8830-e52f952f08a5\") " Oct 30 16:52:11.375580 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.375539 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-credential-socket" pod "a331a4b1-0c84-4850-a289-beda7882adbf" (UID: "a331a4b1-0c84-4850-a289-beda7882adbf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.375580 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.375547 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a331a4b1-0c84-4850-a289-beda7882adbf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a331a4b1-0c84-4850-a289-beda7882adbf-istio-podinfo\") pod \"a331a4b1-0c84-4850-a289-beda7882adbf\" (UID: \"a331a4b1-0c84-4850-a289-beda7882adbf\") " Oct 30 16:52:11.375969 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.375945 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fcac11b-bb6f-4004-8830-e52f952f08a5-istiod-ca-cert" pod "6fcac11b-bb6f-4004-8830-e52f952f08a5" (UID: "6fcac11b-bb6f-4004-8830-e52f952f08a5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:11.376738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376397 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.376738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376424 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.376738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376437 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fcac11b-bb6f-4004-8830-e52f952f08a5-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.376738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376449 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.376738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376461 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.376738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376473 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a331a4b1-0c84-4850-a289-beda7882adbf-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.376738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376485 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.376738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376497 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.376738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376508 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.376738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376730 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-envoy" pod "a331a4b1-0c84-4850-a289-beda7882adbf" (UID: "a331a4b1-0c84-4850-a289-beda7882adbf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.377152 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.376947 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-kube-api-access-6pp79" pod "a331a4b1-0c84-4850-a289-beda7882adbf" (UID: "a331a4b1-0c84-4850-a289-beda7882adbf"). InnerVolumeSpecName "kube-api-access-6pp79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:11.377201 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.377179 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-envoy" pod "6fcac11b-bb6f-4004-8830-e52f952f08a5" (UID: "6fcac11b-bb6f-4004-8830-e52f952f08a5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.377263 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.377234 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-certs" pod "6fcac11b-bb6f-4004-8830-e52f952f08a5" (UID: "6fcac11b-bb6f-4004-8830-e52f952f08a5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.378577 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.378554 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-token" pod "6fcac11b-bb6f-4004-8830-e52f952f08a5" (UID: "6fcac11b-bb6f-4004-8830-e52f952f08a5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:11.379056 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.379026 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a331a4b1-0c84-4850-a289-beda7882adbf-istio-podinfo" pod "a331a4b1-0c84-4850-a289-beda7882adbf" (UID: "a331a4b1-0c84-4850-a289-beda7882adbf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:52:11.379128 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.379053 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-podinfo" pod "6fcac11b-bb6f-4004-8830-e52f952f08a5" (UID: "6fcac11b-bb6f-4004-8830-e52f952f08a5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:52:11.379179 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.379159 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-kube-api-access-mr8tx" pod "6fcac11b-bb6f-4004-8830-e52f952f08a5" (UID: "6fcac11b-bb6f-4004-8830-e52f952f08a5"). InnerVolumeSpecName "kube-api-access-mr8tx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:11.380339 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.380294 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-istio-token" pod "a331a4b1-0c84-4850-a289-beda7882adbf" (UID: "a331a4b1-0c84-4850-a289-beda7882adbf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:11.478514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.478477 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.478514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.478509 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.478514 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.478520 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.478857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.478528 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.478857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.478536 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a331a4b1-0c84-4850-a289-beda7882adbf-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.478857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.478544 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a331a4b1-0c84-4850-a289-beda7882adbf-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.478857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.478552 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fcac11b-bb6f-4004-8830-e52f952f08a5-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.478857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.478560 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pp79\" (UniqueName: \"kubernetes.io/projected/a331a4b1-0c84-4850-a289-beda7882adbf-kube-api-access-6pp79\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.478857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.478570 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mr8tx\" (UniqueName: \"kubernetes.io/projected/6fcac11b-bb6f-4004-8830-e52f952f08a5-kube-api-access-mr8tx\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.689229 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.689148 6400 generic.go:356] "Generic (PLEG): container finished" podID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerID="8a8f273f36689061ad36b51c4b8fad050c198f1dce2d3b539897442af74236a4" exitCode=0 Oct 30 16:52:11.689394 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.689225 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" event={"ID":"0ae9ef48-3347-49a8-b8c9-8a8634d083c1","Type":"ContainerDied","Data":"8a8f273f36689061ad36b51c4b8fad050c198f1dce2d3b539897442af74236a4"} Oct 30 16:52:11.691377 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.691351 6400 generic.go:356] "Generic (PLEG): container finished" podID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerID="f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df" exitCode=0 Oct 30 16:52:11.691523 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.691466 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" event={"ID":"6fcac11b-bb6f-4004-8830-e52f952f08a5","Type":"ContainerDied","Data":"f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df"} Oct 30 16:52:11.691523 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.691501 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" event={"ID":"6fcac11b-bb6f-4004-8830-e52f952f08a5","Type":"ContainerDied","Data":"38c8aa1d32b59a50be2be7acb57ed8a0d54de93f000e973a645405a9bf371695"} Oct 30 16:52:11.691619 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.691523 6400 scope.go:122] "RemoveContainer" containerID="f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df" Oct 30 16:52:11.691690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.691677 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh" Oct 30 16:52:11.697797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.697228 6400 generic.go:356] "Generic (PLEG): container finished" podID="a331a4b1-0c84-4850-a289-beda7882adbf" containerID="51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c" exitCode=0 Oct 30 16:52:11.697797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.697328 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" Oct 30 16:52:11.697797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.697422 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" event={"ID":"a331a4b1-0c84-4850-a289-beda7882adbf","Type":"ContainerDied","Data":"51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c"} Oct 30 16:52:11.697797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.697476 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b" event={"ID":"a331a4b1-0c84-4850-a289-beda7882adbf","Type":"ContainerDied","Data":"2ae2c3b61e352a5b2d48acb3706bae550a5de3f352bf040c36bd6001a7a66167"} Oct 30 16:52:11.711316 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.711158 6400 scope.go:122] "RemoveContainer" containerID="9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b" Oct 30 16:52:11.712866 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.712840 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:52:11.721948 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.721929 6400 scope.go:122] "RemoveContainer" containerID="f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df" Oct 30 16:52:11.722216 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:52:11.722195 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df\": container with ID starting with f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df not found: ID does not exist" containerID="f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df" Oct 30 16:52:11.722282 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.722228 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df"} err="failed to get container status \"f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df\": rpc error: code = NotFound desc = could not find container \"f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df\": container with ID starting with f2da5d1c28892eb11b7ef42aee6850965a38cd5aa7d1e38abe694cbeca4d89df not found: ID does not exist" Oct 30 16:52:11.722282 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.722247 6400 scope.go:122] "RemoveContainer" containerID="9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b" Oct 30 16:52:11.722522 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:52:11.722501 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b\": container with ID starting with 9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b not found: ID does not exist" containerID="9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b" Oct 30 16:52:11.722575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.722529 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b"} err="failed to get container status \"9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b\": rpc error: code = NotFound desc = could not find container \"9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b\": container with ID starting with 9ea782c704d03fcb9468705340b0c0e3f6b626121e847828682ffde68a30582b not found: ID does not exist" Oct 30 16:52:11.722575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.722544 6400 scope.go:122] "RemoveContainer" containerID="51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c" Oct 30 16:52:11.732713 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.732696 6400 scope.go:122] "RemoveContainer" containerID="51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c" Oct 30 16:52:11.733000 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:52:11.732980 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c\": container with ID starting with 51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c not found: ID does not exist" containerID="51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c" Oct 30 16:52:11.733059 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.733008 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c"} err="failed to get container status \"51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c\": rpc error: code = NotFound desc = could not find container \"51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c\": container with ID starting with 51a6a79b2de34f969c05bffe73feb76f91b4b068054025c6c8ff8613ce734c4c not found: ID does not exist" Oct 30 16:52:11.782499 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.782467 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-token\" (UniqueName: \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-token\") pod \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " Oct 30 16:52:11.782707 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.782558 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-credential-socket\") pod \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " Oct 30 16:52:11.782707 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.782617 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istiod-ca-cert\") pod \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " Oct 30 16:52:11.782707 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.782677 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-kube-api-access-6lpcb\" (UniqueName: \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-kube-api-access-6lpcb\") pod \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " Oct 30 16:52:11.782840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.782825 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-certs\") pod \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " Oct 30 16:52:11.782955 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.782936 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-envoy\") pod \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " Oct 30 16:52:11.783031 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.782981 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-data\") pod \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " Oct 30 16:52:11.783078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.783041 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-podinfo\") pod \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " Oct 30 16:52:11.783132 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.783113 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-socket\") pod \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\" (UID: \"0ae9ef48-3347-49a8-b8c9-8a8634d083c1\") " Oct 30 16:52:11.784480 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.784014 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-socket" pod "0ae9ef48-3347-49a8-b8c9-8a8634d083c1" (UID: "0ae9ef48-3347-49a8-b8c9-8a8634d083c1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.784480 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.784123 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istiod-ca-cert" pod "0ae9ef48-3347-49a8-b8c9-8a8634d083c1" (UID: "0ae9ef48-3347-49a8-b8c9-8a8634d083c1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:52:11.784480 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.784149 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-data" pod "0ae9ef48-3347-49a8-b8c9-8a8634d083c1" (UID: "0ae9ef48-3347-49a8-b8c9-8a8634d083c1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.784480 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.784402 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-credential-socket" pod "0ae9ef48-3347-49a8-b8c9-8a8634d083c1" (UID: "0ae9ef48-3347-49a8-b8c9-8a8634d083c1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.785087 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.785065 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-certs" pod "0ae9ef48-3347-49a8-b8c9-8a8634d083c1" (UID: "0ae9ef48-3347-49a8-b8c9-8a8634d083c1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.785343 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.785323 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-token" pod "0ae9ef48-3347-49a8-b8c9-8a8634d083c1" (UID: "0ae9ef48-3347-49a8-b8c9-8a8634d083c1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:11.786641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.786612 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-podinfo" pod "0ae9ef48-3347-49a8-b8c9-8a8634d083c1" (UID: "0ae9ef48-3347-49a8-b8c9-8a8634d083c1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:52:11.786888 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.786868 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-kube-api-access-6lpcb" pod "0ae9ef48-3347-49a8-b8c9-8a8634d083c1" (UID: "0ae9ef48-3347-49a8-b8c9-8a8634d083c1"). InnerVolumeSpecName "kube-api-access-6lpcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:52:11.786967 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.786923 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-envoy" pod "0ae9ef48-3347-49a8-b8c9-8a8634d083c1" (UID: "0ae9ef48-3347-49a8-b8c9-8a8634d083c1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:52:11.800650 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.800573 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b"] Oct 30 16:52:11.806022 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.805996 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-jrq5b"] Oct 30 16:52:11.841147 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.841107 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh"] Oct 30 16:52:11.848004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.847972 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-zq2xh"] Oct 30 16:52:11.884867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.884836 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.884867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.884864 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.884867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.884881 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.885308 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.884891 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.885308 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.884899 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.885308 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.884910 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.885308 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.884918 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.885308 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.884928 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:11.885308 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:11.884937 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lpcb\" (UniqueName: \"kubernetes.io/projected/0ae9ef48-3347-49a8-b8c9-8a8634d083c1-kube-api-access-6lpcb\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:52:12.705384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:12.705357 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" Oct 30 16:52:12.705585 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:12.705357 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4" event={"ID":"0ae9ef48-3347-49a8-b8c9-8a8634d083c1","Type":"ContainerDied","Data":"0e54e019b77cd7f905e85528d964dfe1b29bf09698382552f9ff9b78e43b67a4"} Oct 30 16:52:12.705585 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:12.705498 6400 scope.go:122] "RemoveContainer" containerID="8a8f273f36689061ad36b51c4b8fad050c198f1dce2d3b539897442af74236a4" Oct 30 16:52:12.773355 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:12.773320 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4"] Oct 30 16:52:12.789812 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:12.789771 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-dxcg4"] Oct 30 16:52:12.968825 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:12.968747 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" path="/var/lib/kubelet/pods/0ae9ef48-3347-49a8-b8c9-8a8634d083c1/volumes" Oct 30 16:52:12.969310 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:12.969295 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" path="/var/lib/kubelet/pods/6fcac11b-bb6f-4004-8830-e52f952f08a5/volumes" Oct 30 16:52:12.969762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:12.969749 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" path="/var/lib/kubelet/pods/a331a4b1-0c84-4850-a289-beda7882adbf/volumes" Oct 30 16:52:24.606830 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:52:24.606789 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:52:35.741774 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.741734 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr"] Oct 30 16:52:35.742366 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742350 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" containerName="istio-proxy" Oct 30 16:52:35.742411 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742367 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" containerName="istio-proxy" Oct 30 16:52:35.742411 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742398 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" Oct 30 16:52:35.742411 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742403 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" Oct 30 16:52:35.742511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742412 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" Oct 30 16:52:35.742511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742418 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" Oct 30 16:52:35.742511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742425 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" Oct 30 16:52:35.742511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742429 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" Oct 30 16:52:35.742723 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742516 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a331a4b1-0c84-4850-a289-beda7882adbf" containerName="istio-proxy" Oct 30 16:52:35.742723 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742529 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" Oct 30 16:52:35.742723 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742535 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fcac11b-bb6f-4004-8830-e52f952f08a5" containerName="istio-proxy" Oct 30 16:52:35.742723 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.742540 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae9ef48-3347-49a8-b8c9-8a8634d083c1" containerName="istio-proxy" Oct 30 16:52:35.747646 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.747591 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.766520 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.765447 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-a59502-serving-cert\"" Oct 30 16:52:35.772209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.770323 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr"] Oct 30 16:52:35.843901 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.843862 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-metrics-certs\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.844092 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.843912 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b35351-443b-40e5-b5a4-f548366bba8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.844092 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.844048 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.844197 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.844110 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfwzk\" (UniqueName: \"kubernetes.io/projected/68b35351-443b-40e5-b5a4-f548366bba8e-kube-api-access-bfwzk\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.945230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.945194 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfwzk\" (UniqueName: \"kubernetes.io/projected/68b35351-443b-40e5-b5a4-f548366bba8e-kube-api-access-bfwzk\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.945415 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.945352 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-metrics-certs\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.945415 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.945377 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b35351-443b-40e5-b5a4-f548366bba8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.945491 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.945432 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.946182 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.946152 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b35351-443b-40e5-b5a4-f548366bba8e-openshift-service-ca\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.948160 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.948133 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-metrics-certs\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.948277 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.948260 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-maas-api-tls\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:35.969562 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:35.969526 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfwzk\" (UniqueName: \"kubernetes.io/projected/68b35351-443b-40e5-b5a4-f548366bba8e-kube-api-access-bfwzk\") pod \"maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:36.064082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:36.063999 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:36.386087 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:52:36.386060 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68b35351_443b_40e5_b5a4_f548366bba8e.slice/crio-b57590b287620275402de0e9b28af581073694f5aa3065884681a7ee84762a9d WatchSource:0}: Error finding container b57590b287620275402de0e9b28af581073694f5aa3065884681a7ee84762a9d: Status 404 returned error can't find the container with id b57590b287620275402de0e9b28af581073694f5aa3065884681a7ee84762a9d Oct 30 16:52:36.386443 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:36.386422 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr"] Oct 30 16:52:36.387508 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:36.387492 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:52:36.921914 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:36.921874 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" event={"ID":"68b35351-443b-40e5-b5a4-f548366bba8e","Type":"ContainerStarted","Data":"b57590b287620275402de0e9b28af581073694f5aa3065884681a7ee84762a9d"} Oct 30 16:52:37.929837 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:37.929805 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" event={"ID":"68b35351-443b-40e5-b5a4-f548366bba8e","Type":"ContainerStarted","Data":"ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492"} Oct 30 16:52:37.930363 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:37.929871 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:37.972247 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:37.972195 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" podStartSLOduration=2.566728384 podStartE2EDuration="2.972181046s" totalImagesPullingTime="405.452662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:37.969994201 +0000 UTC m=+1867.120275202" watchObservedRunningTime="2026-10-30 16:52:37.972181046 +0000 UTC m=+1867.122462034" Oct 30 16:52:43.950205 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:43.950170 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:52:49.238931 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.238897 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc"] Oct 30 16:52:49.243724 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.243699 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.276176 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.276145 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-17f60f-serving-cert\"" Oct 30 16:52:49.281638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.281594 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc"] Oct 30 16:52:49.309135 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.309101 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-metrics-certs\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.309135 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.309136 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440019cb-b006-43dc-b146-291a66ec787b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.309393 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.309239 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-maas-api-tls\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.309503 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.309474 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvptg\" (UniqueName: \"kubernetes.io/projected/440019cb-b006-43dc-b146-291a66ec787b-kube-api-access-jvptg\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.411265 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.411226 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvptg\" (UniqueName: \"kubernetes.io/projected/440019cb-b006-43dc-b146-291a66ec787b-kube-api-access-jvptg\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.411461 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.411360 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-metrics-certs\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.411461 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.411378 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440019cb-b006-43dc-b146-291a66ec787b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.411461 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.411415 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-maas-api-tls\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.411670 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:52:49.411649 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-serving-cert: secret "maas-api-e2e-inf-a-17f60f-serving-cert" not found Oct 30 16:52:49.411744 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:52:49.411734 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-maas-api-tls podName:440019cb-b006-43dc-b146-291a66ec787b nodeName:}" failed. No retries permitted until 2026-10-30 16:52:49.911715732 +0000 UTC m=+1879.061996723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-maas-api-tls") pod "maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" (UID: "440019cb-b006-43dc-b146-291a66ec787b") : secret "maas-api-e2e-inf-a-17f60f-serving-cert" not found Oct 30 16:52:49.412144 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.412127 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440019cb-b006-43dc-b146-291a66ec787b-openshift-service-ca\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.413977 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.413960 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-metrics-certs\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.437561 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.437529 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvptg\" (UniqueName: \"kubernetes.io/projected/440019cb-b006-43dc-b146-291a66ec787b-kube-api-access-jvptg\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.921617 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.921575 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-maas-api-tls\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:49.924340 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:49.924312 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-maas-api-tls\") pod \"maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:50.160098 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:50.160060 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:50.508029 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:52:50.508000 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod440019cb_b006_43dc_b146_291a66ec787b.slice/crio-c225363ecd1dfba036ab19858a7c14f86b666bd9a414103f77d4ad1b90ca15c1 WatchSource:0}: Error finding container c225363ecd1dfba036ab19858a7c14f86b666bd9a414103f77d4ad1b90ca15c1: Status 404 returned error can't find the container with id c225363ecd1dfba036ab19858a7c14f86b666bd9a414103f77d4ad1b90ca15c1 Oct 30 16:52:50.508519 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:50.508495 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc"] Oct 30 16:52:51.171382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:51.171284 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" event={"ID":"440019cb-b006-43dc-b146-291a66ec787b","Type":"ContainerStarted","Data":"2171dc0597aca21fd700997a235e3a9610b0251a4bcac2161059c97fdeacb327"} Oct 30 16:52:51.171694 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:51.171670 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" event={"ID":"440019cb-b006-43dc-b146-291a66ec787b","Type":"ContainerStarted","Data":"c225363ecd1dfba036ab19858a7c14f86b666bd9a414103f77d4ad1b90ca15c1"} Oct 30 16:52:51.172990 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:51.172969 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:52:51.234902 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:51.234838 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" podStartSLOduration=1.924626946 podStartE2EDuration="2.234737658s" totalImagesPullingTime="310.110712ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:52:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:52:51.234157065 +0000 UTC m=+1880.384438054" watchObservedRunningTime="2026-10-30 16:52:51.234737658 +0000 UTC m=+1880.385018647" Oct 30 16:52:54.601298 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:52:54.601255 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:52:58.202812 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:52:58.202779 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 16:53:05.163463 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.163429 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv"] Oct 30 16:53:05.167849 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.167831 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.183175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.183147 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-3ed10a-serving-cert\"" Oct 30 16:53:05.200235 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.200202 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv"] Oct 30 16:53:05.277868 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.277823 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba32efe-e717-4821-996d-9da99af18277-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.278078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.278008 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.278078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.278049 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.278222 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.278199 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmrf5\" (UniqueName: \"kubernetes.io/projected/bba32efe-e717-4821-996d-9da99af18277-kube-api-access-vmrf5\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.379970 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.379933 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.380157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.379987 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.380157 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.380051 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmrf5\" (UniqueName: \"kubernetes.io/projected/bba32efe-e717-4821-996d-9da99af18277-kube-api-access-vmrf5\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.380249 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.380198 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba32efe-e717-4821-996d-9da99af18277-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.380854 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.380828 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba32efe-e717-4821-996d-9da99af18277-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.382831 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.382805 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.383009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.382988 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.403043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.403016 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmrf5\" (UniqueName: \"kubernetes.io/projected/bba32efe-e717-4821-996d-9da99af18277-kube-api-access-vmrf5\") pod \"maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.484859 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.484831 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:05.951503 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:53:05.951468 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba32efe_e717_4821_996d_9da99af18277.slice/crio-18991ab2473da218f2dde305502bd24589a82194ceddf6ab623e766a52a8076d WatchSource:0}: Error finding container 18991ab2473da218f2dde305502bd24589a82194ceddf6ab623e766a52a8076d: Status 404 returned error can't find the container with id 18991ab2473da218f2dde305502bd24589a82194ceddf6ab623e766a52a8076d Oct 30 16:53:05.952158 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:05.952134 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv"] Oct 30 16:53:06.293525 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:06.293492 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" event={"ID":"bba32efe-e717-4821-996d-9da99af18277","Type":"ContainerStarted","Data":"18991ab2473da218f2dde305502bd24589a82194ceddf6ab623e766a52a8076d"} Oct 30 16:53:07.309768 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:07.309726 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" event={"ID":"bba32efe-e717-4821-996d-9da99af18277","Type":"ContainerStarted","Data":"e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888"} Oct 30 16:53:07.310199 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:07.309837 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:07.385957 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:07.385905 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" podStartSLOduration=2.046162529 podStartE2EDuration="2.385888949s" totalImagesPullingTime="339.72642ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:07.383436025 +0000 UTC m=+1896.533717013" watchObservedRunningTime="2026-10-30 16:53:07.385888949 +0000 UTC m=+1896.536169936" Oct 30 16:53:13.322456 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:13.322427 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 16:53:24.602419 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:53:24.602376 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:53:24.970761 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:24.970686 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt"] Oct 30 16:53:24.975354 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:24.975325 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:24.995782 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:24.995753 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt"] Oct 30 16:53:25.002689 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.002662 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-480a60\"/\"openshift-service-ca.crt\"" Oct 30 16:53:25.002878 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.002674 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-480a60\"/\"default-dockercfg-4rkh6\"" Oct 30 16:53:25.002878 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.002691 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-480a60\"/\"kube-root-ca.crt\"" Oct 30 16:53:25.002878 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.002764 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-480a60\"/\"test-model-480a60-kserve-self-signed-certs\"" Oct 30 16:53:25.132299 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.132263 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-model-cache\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.132516 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.132417 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9r2wz\" (UniqueName: \"kubernetes.io/projected/2530a183-f3eb-4326-8702-14a1f7492d17-kube-api-access-9r2wz\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.132516 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.132444 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2530a183-f3eb-4326-8702-14a1f7492d17-tls-certs\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.132516 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.132468 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-tmp-dir\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.132696 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.132678 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-dshm\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.132746 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.132727 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-home\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.238633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.237705 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-home\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.238633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.237975 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-model-cache\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.238633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.238104 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9r2wz\" (UniqueName: \"kubernetes.io/projected/2530a183-f3eb-4326-8702-14a1f7492d17-kube-api-access-9r2wz\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.238633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.238132 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2530a183-f3eb-4326-8702-14a1f7492d17-tls-certs\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.238633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.238157 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-tmp-dir\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.238633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.238305 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-dshm\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.243638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.239435 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-home\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.243638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.239707 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-model-cache\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.243638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.239935 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-tmp-dir\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.251529 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.251495 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2530a183-f3eb-4326-8702-14a1f7492d17-tls-certs\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.252016 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.251994 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-dshm\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.273592 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.263174 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9r2wz\" (UniqueName: \"kubernetes.io/projected/2530a183-f3eb-4326-8702-14a1f7492d17-kube-api-access-9r2wz\") pod \"test-model-480a60-kserve-5794d9c9b5-twtrt\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.298917 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.298883 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:25.658166 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:53:25.655408 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2530a183_f3eb_4326_8702_14a1f7492d17.slice/crio-a50005abade8371d4f55f316a3d64cfad7c6a6a78f446367b71b15cebb4e72b2 WatchSource:0}: Error finding container a50005abade8371d4f55f316a3d64cfad7c6a6a78f446367b71b15cebb4e72b2: Status 404 returned error can't find the container with id a50005abade8371d4f55f316a3d64cfad7c6a6a78f446367b71b15cebb4e72b2 Oct 30 16:53:25.658166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:25.656098 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt"] Oct 30 16:53:26.455442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:26.455406 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" event={"ID":"2530a183-f3eb-4326-8702-14a1f7492d17","Type":"ContainerStarted","Data":"68670bf50f362d7d4d8563da7a6fb236426e2ab81cd5c011751859a045ee6080"} Oct 30 16:53:26.455694 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:26.455456 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" event={"ID":"2530a183-f3eb-4326-8702-14a1f7492d17","Type":"ContainerStarted","Data":"a50005abade8371d4f55f316a3d64cfad7c6a6a78f446367b71b15cebb4e72b2"} Oct 30 16:53:26.496074 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:26.496005 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" podStartSLOduration=2.4959918500000002 podStartE2EDuration="2.49599185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:26.495198977 +0000 UTC m=+1915.645479980" watchObservedRunningTime="2026-10-30 16:53:26.49599185 +0000 UTC m=+1915.646272837" Oct 30 16:53:27.300082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:27.300043 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:35.300136 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:35.300094 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:35.314551 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:35.314199 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:35.542075 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:35.542044 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:53:37.778738 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.778703 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv"] Oct 30 16:53:37.784445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.784421 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.813408 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.813362 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-668047\"/\"kube-root-ca.crt\"" Oct 30 16:53:37.814883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.814253 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv"] Oct 30 16:53:37.816445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.816108 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-668047\"/\"default-dockercfg-fpbzn\"" Oct 30 16:53:37.816445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.816228 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-668047\"/\"test-model-668047-kserve-self-signed-certs\"" Oct 30 16:53:37.817105 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.817084 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-668047\"/\"openshift-service-ca.crt\"" Oct 30 16:53:37.893359 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.893314 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-dshm\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.893579 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.893447 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-model-cache\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.893579 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.893483 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tmp-dir\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.893579 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.893562 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tls-certs\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.893788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.893586 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fhcv\" (UniqueName: \"kubernetes.io/projected/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-kube-api-access-2fhcv\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.893788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.893713 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-home\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.995269 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.995229 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-dshm\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.995476 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.995310 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-model-cache\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.995476 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.995335 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tmp-dir\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.995476 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.995366 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tls-certs\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.995476 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.995388 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fhcv\" (UniqueName: \"kubernetes.io/projected/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-kube-api-access-2fhcv\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.995794 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.995769 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-home\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.995860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.995819 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tmp-dir\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.995923 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.995869 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-model-cache\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.996129 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.996105 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-home\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.998184 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.998151 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-dshm\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:37.998351 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:37.998335 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tls-certs\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:38.019445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:38.019416 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fhcv\" (UniqueName: \"kubernetes.io/projected/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-kube-api-access-2fhcv\") pod \"test-model-668047-kserve-6f45d6bd77-j8zjv\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:38.104688 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:38.104080 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:38.522079 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:38.522046 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv"] Oct 30 16:53:38.522839 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:53:38.522810 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbda2dd6e_eacb_429c_aac2_982f3d1e8ade.slice/crio-47cd34f1669b2b29f465464512a15ebf3de039a01759fe19a5afc7c11b7db362 WatchSource:0}: Error finding container 47cd34f1669b2b29f465464512a15ebf3de039a01759fe19a5afc7c11b7db362: Status 404 returned error can't find the container with id 47cd34f1669b2b29f465464512a15ebf3de039a01759fe19a5afc7c11b7db362 Oct 30 16:53:38.551281 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:38.551247 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" event={"ID":"bda2dd6e-eacb-429c-aac2-982f3d1e8ade","Type":"ContainerStarted","Data":"47cd34f1669b2b29f465464512a15ebf3de039a01759fe19a5afc7c11b7db362"} Oct 30 16:53:39.561689 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:39.561646 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" event={"ID":"bda2dd6e-eacb-429c-aac2-982f3d1e8ade","Type":"ContainerStarted","Data":"9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd"} Oct 30 16:53:39.605719 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:39.605658 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" podStartSLOduration=2.605639808 podStartE2EDuration="2.605639808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:53:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:53:39.602055378 +0000 UTC m=+1928.752336369" watchObservedRunningTime="2026-10-30 16:53:39.605639808 +0000 UTC m=+1928.755920796" Oct 30 16:53:40.105494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:40.105459 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:48.105502 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:48.105454 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:48.119715 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:48.119688 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:48.702935 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:53:48.702904 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:53:54.608896 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:53:54.608847 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:54:09.394052 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.393999 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm"] Oct 30 16:54:09.399100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.399061 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.426437 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.426404 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9bfcac\"/\"openshift-service-ca.crt\"" Oct 30 16:54:09.426437 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.426424 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9bfcac\"/\"default-dockercfg-l66t9\"" Oct 30 16:54:09.426775 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.426459 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-9bfcac\"/\"kube-root-ca.crt\"" Oct 30 16:54:09.426775 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.426645 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-9bfcac\"/\"test-model-9bfcac-kserve-self-signed-certs\"" Oct 30 16:54:09.430332 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.430306 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm"] Oct 30 16:54:09.540287 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.540246 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-dshm\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.540501 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.540339 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-home\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.540501 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.540425 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-model-cache\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.540501 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.540498 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-tmp-dir\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.540668 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.540584 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jtqg\" (UniqueName: \"kubernetes.io/projected/97637696-5890-4712-b677-f9ef36fdf467-kube-api-access-6jtqg\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.540716 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.540694 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97637696-5890-4712-b677-f9ef36fdf467-tls-certs\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.643163 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.643127 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-tmp-dir\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.643446 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.643424 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jtqg\" (UniqueName: \"kubernetes.io/projected/97637696-5890-4712-b677-f9ef36fdf467-kube-api-access-6jtqg\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.643641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.643622 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97637696-5890-4712-b677-f9ef36fdf467-tls-certs\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.643980 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.643958 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-dshm\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.644181 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.644133 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-home\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.644316 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.644300 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-model-cache\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.644904 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.644873 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-model-cache\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.645303 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.645281 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-tmp-dir\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.646720 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.646699 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-home\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.716689 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.685523 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97637696-5890-4712-b677-f9ef36fdf467-tls-certs\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.766702 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.707587 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jtqg\" (UniqueName: \"kubernetes.io/projected/97637696-5890-4712-b677-f9ef36fdf467-kube-api-access-6jtqg\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:09.766975 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:09.713559 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-dshm\") pod \"test-model-9bfcac-kserve-fb56cbb95-276pm\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:10.015998 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:10.015961 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:10.333596 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:54:10.333563 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod97637696_5890_4712_b677_f9ef36fdf467.slice/crio-96681b616564e44804ec54c4c65e37188f83bcc87027f8a6b19fc954b0e51639 WatchSource:0}: Error finding container 96681b616564e44804ec54c4c65e37188f83bcc87027f8a6b19fc954b0e51639: Status 404 returned error can't find the container with id 96681b616564e44804ec54c4c65e37188f83bcc87027f8a6b19fc954b0e51639 Oct 30 16:54:10.333734 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:10.333648 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm"] Oct 30 16:54:11.131114 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:11.131082 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" event={"ID":"97637696-5890-4712-b677-f9ef36fdf467","Type":"ContainerStarted","Data":"46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c"} Oct 30 16:54:11.131540 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:11.131124 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" event={"ID":"97637696-5890-4712-b677-f9ef36fdf467","Type":"ContainerStarted","Data":"96681b616564e44804ec54c4c65e37188f83bcc87027f8a6b19fc954b0e51639"} Oct 30 16:54:11.172516 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:11.172456 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" podStartSLOduration=2.172440647 podStartE2EDuration="2.172440647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:11.171005409 +0000 UTC m=+1960.321286398" watchObservedRunningTime="2026-10-30 16:54:11.172440647 +0000 UTC m=+1960.322721635" Oct 30 16:54:12.017124 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:12.017090 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:20.016509 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:20.016468 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:20.029675 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:20.029646 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:20.207913 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:20.207882 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:54:24.601637 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:54:24.601585 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:54:27.806638 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:27.804055 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw"] Oct 30 16:54:27.809191 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:27.809162 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:27.836536 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:27.836503 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-9c1e26-serving-cert\"" Oct 30 16:54:27.840894 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:27.840852 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw"] Oct 30 16:54:27.973641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:27.973588 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:27.973860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:27.973834 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-metrics-certs\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:27.973920 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:27.973898 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfw85\" (UniqueName: \"kubernetes.io/projected/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-kube-api-access-qfw85\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:27.974032 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:27.974011 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:28.075666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:28.075575 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-metrics-certs\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:28.075853 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:28.075699 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfw85\" (UniqueName: \"kubernetes.io/projected/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-kube-api-access-qfw85\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:28.075853 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:28.075805 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:28.075981 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:28.075902 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:28.076334 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:54:28.076308 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-serving-cert: secret "maas-api-e2e-inf-b-9c1e26-serving-cert" not found Oct 30 16:54:28.076665 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:54:28.076375 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-maas-api-tls podName:eabab492-b8ae-4c10-8c3f-d82e63c85ee1 nodeName:}" failed. No retries permitted until 2026-10-30 16:54:28.576359872 +0000 UTC m=+1977.726640840 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-maas-api-tls") pod "maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" (UID: "eabab492-b8ae-4c10-8c3f-d82e63c85ee1") : secret "maas-api-e2e-inf-b-9c1e26-serving-cert" not found Oct 30 16:54:28.076875 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:28.076842 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-openshift-service-ca\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:28.079500 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:28.079479 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-metrics-certs\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:28.101593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:28.101521 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfw85\" (UniqueName: \"kubernetes.io/projected/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-kube-api-access-qfw85\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:28.582534 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:28.582494 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:28.585213 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:28.585184 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-maas-api-tls\") pod \"maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:28.727483 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:28.727447 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:29.057045 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:54:29.056999 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeabab492_b8ae_4c10_8c3f_d82e63c85ee1.slice/crio-b47e6416fd16fa30d5d02c2ad1935df72483e93932399342da88f7c03c948444 WatchSource:0}: Error finding container b47e6416fd16fa30d5d02c2ad1935df72483e93932399342da88f7c03c948444: Status 404 returned error can't find the container with id b47e6416fd16fa30d5d02c2ad1935df72483e93932399342da88f7c03c948444 Oct 30 16:54:29.057849 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:29.057824 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw"] Oct 30 16:54:29.316350 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:29.316242 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" event={"ID":"eabab492-b8ae-4c10-8c3f-d82e63c85ee1","Type":"ContainerStarted","Data":"b47e6416fd16fa30d5d02c2ad1935df72483e93932399342da88f7c03c948444"} Oct 30 16:54:30.325445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:30.325408 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" event={"ID":"eabab492-b8ae-4c10-8c3f-d82e63c85ee1","Type":"ContainerStarted","Data":"7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7"} Oct 30 16:54:30.325888 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:30.325524 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:30.366710 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:30.366654 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" podStartSLOduration=2.988880104 podStartE2EDuration="3.366635555s" totalImagesPullingTime="377.755451ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:30.366118697 +0000 UTC m=+1979.516399686" watchObservedRunningTime="2026-10-30 16:54:30.366635555 +0000 UTC m=+1979.516916536" Oct 30 16:54:36.338495 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:36.338464 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 16:54:40.320748 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.320710 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft"] Oct 30 16:54:40.450710 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.426766 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.450710 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.442459 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-689899-serving-cert\"" Oct 30 16:54:40.470232 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.470198 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft"] Oct 30 16:54:40.526532 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.526489 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31009489-863e-4dad-9818-d662fa62d429-openshift-service-ca\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.526788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.526567 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-maas-api-tls\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.526788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.526623 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-metrics-certs\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.526788 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.526778 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6np5\" (UniqueName: \"kubernetes.io/projected/31009489-863e-4dad-9818-d662fa62d429-kube-api-access-k6np5\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.629072 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.628964 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31009489-863e-4dad-9818-d662fa62d429-openshift-service-ca\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.629072 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.629044 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-maas-api-tls\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.629072 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.629073 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-metrics-certs\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.629312 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:54:40.629206 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-serving-cert: secret "maas-api-e2e-inf-b-689899-serving-cert" not found Oct 30 16:54:40.629312 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:54:40.629296 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-maas-api-tls podName:31009489-863e-4dad-9818-d662fa62d429 nodeName:}" failed. No retries permitted until 2026-10-30 16:54:41.12927415 +0000 UTC m=+1990.279555136 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-maas-api-tls") pod "maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" (UID: "31009489-863e-4dad-9818-d662fa62d429") : secret "maas-api-e2e-inf-b-689899-serving-cert" not found Oct 30 16:54:40.629312 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.629214 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6np5\" (UniqueName: \"kubernetes.io/projected/31009489-863e-4dad-9818-d662fa62d429-kube-api-access-k6np5\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.629815 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.629776 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31009489-863e-4dad-9818-d662fa62d429-openshift-service-ca\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.632372 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.632348 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-metrics-certs\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:40.659905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:40.659872 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6np5\" (UniqueName: \"kubernetes.io/projected/31009489-863e-4dad-9818-d662fa62d429-kube-api-access-k6np5\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:41.140631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:41.138081 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-maas-api-tls\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:41.150272 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:41.150235 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-maas-api-tls\") pod \"maas-api-e2e-inf-b-689899-55ccb9d775-xjqft\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:41.396002 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:41.395922 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:41.710943 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:54:41.710910 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31009489_863e_4dad_9818_d662fa62d429.slice/crio-b7df83b33f33d4c0ac3885d9e7e51d7acdafaba446dfbaa42534d3aae3f35149 WatchSource:0}: Error finding container b7df83b33f33d4c0ac3885d9e7e51d7acdafaba446dfbaa42534d3aae3f35149: Status 404 returned error can't find the container with id b7df83b33f33d4c0ac3885d9e7e51d7acdafaba446dfbaa42534d3aae3f35149 Oct 30 16:54:41.712296 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:41.712271 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft"] Oct 30 16:54:42.653347 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:42.653309 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" event={"ID":"31009489-863e-4dad-9818-d662fa62d429","Type":"ContainerStarted","Data":"a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73"} Oct 30 16:54:42.653797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:42.653373 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" event={"ID":"31009489-863e-4dad-9818-d662fa62d429","Type":"ContainerStarted","Data":"b7df83b33f33d4c0ac3885d9e7e51d7acdafaba446dfbaa42534d3aae3f35149"} Oct 30 16:54:42.653797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:42.653403 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:42.697634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:42.697567 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" podStartSLOduration=2.371584377 podStartE2EDuration="2.697551979s" totalImagesPullingTime="325.967602ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:42.693985687 +0000 UTC m=+1991.844266676" watchObservedRunningTime="2026-10-30 16:54:42.697551979 +0000 UTC m=+1991.847832966" Oct 30 16:54:48.668897 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:48.668856 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 16:54:50.160281 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.160243 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd"] Oct 30 16:54:50.164866 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.164842 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.183992 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.183955 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-7c4b84-serving-cert\"" Oct 30 16:54:50.191813 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.191782 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd"] Oct 30 16:54:50.193153 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.192769 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.193547 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.193518 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.193712 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.193659 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9449l\" (UniqueName: \"kubernetes.io/projected/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-kube-api-access-9449l\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.193881 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.193862 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.295493 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.295462 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.295710 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.295516 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9449l\" (UniqueName: \"kubernetes.io/projected/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-kube-api-access-9449l\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.295710 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.295694 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.295858 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.295782 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.295858 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:54:50.295818 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-serving-cert: secret "maas-api-e2e-shared-b-w4-7c4b84-serving-cert" not found Oct 30 16:54:50.296005 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:54:50.295897 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-maas-api-tls podName:a7e415d4-b0b2-40c1-b9fe-beb23f9c0809 nodeName:}" failed. No retries permitted until 2026-10-30 16:54:50.795869877 +0000 UTC m=+1999.946150860 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-maas-api-tls") pod "maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" (UID: "a7e415d4-b0b2-40c1-b9fe-beb23f9c0809") : secret "maas-api-e2e-shared-b-w4-7c4b84-serving-cert" not found Oct 30 16:54:50.296415 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.296395 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.298381 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.298349 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.321903 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.321875 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9449l\" (UniqueName: \"kubernetes.io/projected/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-kube-api-access-9449l\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.803138 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.803090 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:50.805802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:50.805778 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:51.085253 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:51.085160 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:51.409949 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:51.409917 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd"] Oct 30 16:54:51.427626 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:54:51.413877 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7e415d4_b0b2_40c1_b9fe_beb23f9c0809.slice/crio-1ce9e1af2fb8a848ec2078b60482ee357040997c266a2a1d0e01f6c607a1e94b WatchSource:0}: Error finding container 1ce9e1af2fb8a848ec2078b60482ee357040997c266a2a1d0e01f6c607a1e94b: Status 404 returned error can't find the container with id 1ce9e1af2fb8a848ec2078b60482ee357040997c266a2a1d0e01f6c607a1e94b Oct 30 16:54:51.726089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:51.726001 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" event={"ID":"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809","Type":"ContainerStarted","Data":"1ce9e1af2fb8a848ec2078b60482ee357040997c266a2a1d0e01f6c607a1e94b"} Oct 30 16:54:52.740285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:52.740250 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" event={"ID":"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809","Type":"ContainerStarted","Data":"753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0"} Oct 30 16:54:52.740811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:52.740380 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:54:52.784425 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:52.784372 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" podStartSLOduration=2.411504199 podStartE2EDuration="2.7843586s" totalImagesPullingTime="372.854401ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:54:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:54:52.782714366 +0000 UTC m=+2001.932995354" watchObservedRunningTime="2026-10-30 16:54:52.7843586 +0000 UTC m=+2001.934639617" Oct 30 16:54:54.603443 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:54:54.603405 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:54:58.754175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:54:58.754146 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 16:55:18.306314 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.306282 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp"] Oct 30 16:55:18.311951 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.311926 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.344633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.344586 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp"] Oct 30 16:55:18.411216 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.411186 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-689899-openshift-default-dockercfg-x7vsc\"" Oct 30 16:55:18.485043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.485006 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.485222 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.485062 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07688b22-fe9c-49b1-9436-bd8f642cae88-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.485313 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.485225 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqfj\" (UniqueName: \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-kube-api-access-shqfj\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.485313 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.485279 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.485475 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.485458 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.485512 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.485489 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.485653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.485638 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.485721 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.485679 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.485797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.485759 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.587576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.587483 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.587576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.587546 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.587867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.587595 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.587867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.587650 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.587867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.587692 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07688b22-fe9c-49b1-9436-bd8f642cae88-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.587867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.587787 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqfj\" (UniqueName: \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-kube-api-access-shqfj\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.587867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.587836 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.588150 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.587983 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.588150 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.588020 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.588150 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.588070 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.588150 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.588104 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.588480 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.588457 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.588565 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.588529 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.588667 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.588590 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07688b22-fe9c-49b1-9436-bd8f642cae88-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.591008 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.590973 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.591241 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.591221 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.612576 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.612545 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.612759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.612705 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqfj\" (UniqueName: \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-kube-api-access-shqfj\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.630950 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.630923 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:18.970325 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:55:18.970299 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07688b22_fe9c_49b1_9436_bd8f642cae88.slice/crio-b2274beff607643ebb529ef8c9bd1ae6b9c75c0c0c54d01c4bae9be0a8aa9f3c WatchSource:0}: Error finding container b2274beff607643ebb529ef8c9bd1ae6b9c75c0c0c54d01c4bae9be0a8aa9f3c: Status 404 returned error can't find the container with id b2274beff607643ebb529ef8c9bd1ae6b9c75c0c0c54d01c4bae9be0a8aa9f3c Oct 30 16:55:18.971125 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.971093 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp"] Oct 30 16:55:18.973404 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.973363 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:18.973498 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.973441 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:18.973498 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.973472 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:18.996759 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:18.996729 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" event={"ID":"07688b22-fe9c-49b1-9436-bd8f642cae88","Type":"ContainerStarted","Data":"b2274beff607643ebb529ef8c9bd1ae6b9c75c0c0c54d01c4bae9be0a8aa9f3c"} Oct 30 16:55:20.005415 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:20.005375 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" event={"ID":"07688b22-fe9c-49b1-9436-bd8f642cae88","Type":"ContainerStarted","Data":"0276e06a14430bc2e2d665d10a63aef20feb233d546582242a58cb529f1a4d90"} Oct 30 16:55:20.063233 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:20.063165 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" podStartSLOduration=2.063151201 podStartE2EDuration="2.063151201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:20.060507176 +0000 UTC m=+2029.210788164" watchObservedRunningTime="2026-10-30 16:55:20.063151201 +0000 UTC m=+2029.213432188" Oct 30 16:55:20.635699 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:20.635652 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:20.636268 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:20.636237 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 16:55:20.636485 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:20.636454 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 16:55:21.632637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:21.631788 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 16:55:21.632637 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:21.631856 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 16:55:22.632628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:22.632257 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 16:55:22.632628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:22.632318 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 16:55:23.635054 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:23.635018 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 16:55:23.635735 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:23.635701 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 16:55:24.605002 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:55:24.604964 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:55:24.633001 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:24.632959 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 16:55:24.633278 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:24.633246 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 16:55:25.631811 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:25.631774 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" start-of-body= Oct 30 16:55:25.632338 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:25.631828 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.128:15021/healthz/ready\": dial tcp 10.134.0.128:15021: connect: connection refused" Oct 30 16:55:26.636716 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:26.636685 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:26.637173 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:26.637117 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:26.637890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:26.637872 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 16:55:48.381797 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.381766 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb"] Oct 30 16:55:48.387852 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.387824 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.413083 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.413051 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb"] Oct 30 16:55:48.464836 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.464793 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.465070 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.464902 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.465220 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.465158 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.465334 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.465293 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.465426 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.465361 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nlkw\" (UniqueName: \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-kube-api-access-7nlkw\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.465426 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.465392 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.465665 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.465512 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.465665 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.465535 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.465665 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.465567 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7761f76-bf36-446a-a128-74ac9fa1a1da-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.566804 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.566767 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.566984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.566849 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.566984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.566904 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.566984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.566946 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nlkw\" (UniqueName: \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-kube-api-access-7nlkw\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.566984 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.566969 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.567197 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.567119 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.567197 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.567156 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.567317 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.567195 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7761f76-bf36-446a-a128-74ac9fa1a1da-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.567317 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.567221 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.567317 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.567283 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.567486 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.567349 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.567486 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.567383 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.567486 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.567447 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.568154 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.568115 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7761f76-bf36-446a-a128-74ac9fa1a1da-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.569656 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.569633 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.570371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.570351 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.590331 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.590307 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.590535 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.590510 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nlkw\" (UniqueName: \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-kube-api-access-7nlkw\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:48.704166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:48.704080 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:49.142137 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:49.137254 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb"] Oct 30 16:55:49.144666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:49.143543 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:49.144666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:49.143671 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:49.144666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:49.143722 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:55:49.380243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:49.380109 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" event={"ID":"b7761f76-bf36-446a-a128-74ac9fa1a1da","Type":"ContainerStarted","Data":"f21a0f253915798e90198f91b694bf06895c0da497d14e63fec9474e49c3df0c"} Oct 30 16:55:50.388660 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:50.388623 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" event={"ID":"b7761f76-bf36-446a-a128-74ac9fa1a1da","Type":"ContainerStarted","Data":"3e7e4515450bbf17a0ee727edeae6d391da7f285ee24cfbff79b5d4b706d67a0"} Oct 30 16:55:50.460258 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:50.460195 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" podStartSLOduration=2.460180421 podStartE2EDuration="2.460180421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:55:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:55:50.457295059 +0000 UTC m=+2059.607576047" watchObservedRunningTime="2026-10-30 16:55:50.460180421 +0000 UTC m=+2059.610461409" Oct 30 16:55:50.705298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:50.705217 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:50.706850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:50.706824 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 16:55:50.706982 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:50.706882 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 16:55:51.704861 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:51.704829 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 16:55:51.705282 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:51.704899 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 16:55:52.704488 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:52.704446 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 16:55:52.704696 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:52.704506 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 16:55:53.705302 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:53.705263 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 16:55:53.705737 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:53.705329 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 16:55:54.605101 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:55:54.605060 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:55:54.705380 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:54.705341 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 16:55:54.705828 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:54.705405 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 16:55:55.708631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:55.707940 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" start-of-body= Oct 30 16:55:55.708631 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:55.708012 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.129:15021/healthz/ready\": dial tcp 10.134.0.129:15021: connect: connection refused" Oct 30 16:55:56.711063 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:56.711034 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:56.711555 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:56.711535 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:55:56.712302 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:55:56.712281 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 16:56:13.146662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.146557 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965"] Oct 30 16:56:13.151333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.151299 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.178830 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.178801 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3ed10a\"/\"openshift-service-ca.crt\"" Oct 30 16:56:13.179035 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.179007 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3ed10a\"/\"auth-test-model-3ed10a-kserve-self-signed-certs\"" Oct 30 16:56:13.179035 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.178987 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3ed10a\"/\"kube-root-ca.crt\"" Oct 30 16:56:13.179243 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.179072 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3ed10a\"/\"default-dockercfg-jg82r\"" Oct 30 16:56:13.183699 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.183676 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965"] Oct 30 16:56:13.219938 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.219895 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tls-certs\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.220132 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.220037 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fzn2t\" (UniqueName: \"kubernetes.io/projected/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-kube-api-access-fzn2t\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.220132 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.220068 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-model-cache\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.220238 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.220161 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-dshm\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.220298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.220255 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-home\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.220353 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.220312 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tmp-dir\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.321789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.321761 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tls-certs\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.321947 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.321846 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fzn2t\" (UniqueName: \"kubernetes.io/projected/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-kube-api-access-fzn2t\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.321947 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.321866 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-model-cache\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.321947 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.321896 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-dshm\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.322109 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.321994 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-home\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.322109 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.322017 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tmp-dir\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.322847 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.322818 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-home\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.322996 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.322916 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tmp-dir\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.323164 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.323143 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-model-cache\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.324657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.324575 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-dshm\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.324657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.324623 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tls-certs\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.347872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.347837 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fzn2t\" (UniqueName: \"kubernetes.io/projected/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-kube-api-access-fzn2t\") pod \"auth-test-model-3ed10a-kserve-7b6854478f-zd965\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.479474 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.479386 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:13.865540 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:56:13.865513 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84b45e9d_16ec_4b55_a535_2b9fb9ac2137.slice/crio-0074fd02adbb40b5fae83c9d33874940ef2f14dcd541d345d2a877bef22b3c80 WatchSource:0}: Error finding container 0074fd02adbb40b5fae83c9d33874940ef2f14dcd541d345d2a877bef22b3c80: Status 404 returned error can't find the container with id 0074fd02adbb40b5fae83c9d33874940ef2f14dcd541d345d2a877bef22b3c80 Oct 30 16:56:13.865715 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:13.865643 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965"] Oct 30 16:56:14.667619 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:14.667571 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" event={"ID":"84b45e9d-16ec-4b55-a535-2b9fb9ac2137","Type":"ContainerStarted","Data":"b5fd342751fa3d52bd899d6f0b63259bb2ee09bfe0a87eba8e9288fd7be5153c"} Oct 30 16:56:14.668082 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:14.667659 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" event={"ID":"84b45e9d-16ec-4b55-a535-2b9fb9ac2137","Type":"ContainerStarted","Data":"0074fd02adbb40b5fae83c9d33874940ef2f14dcd541d345d2a877bef22b3c80"} Oct 30 16:56:14.709816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:14.709757 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" podStartSLOduration=1.7097430789999999 podStartE2EDuration="1.709743079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:56:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:56:14.706856954 +0000 UTC m=+2083.857137933" watchObservedRunningTime="2026-10-30 16:56:14.709743079 +0000 UTC m=+2083.860024067" Oct 30 16:56:15.480508 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:15.480480 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:18.537822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:18.537784 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt"] Oct 30 16:56:23.482756 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:23.482678 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:23.529488 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:23.529459 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:23.757441 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:23.757359 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 16:56:24.601620 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:56:24.601560 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:56:33.567816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.567764 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" podUID="2530a183-f3eb-4326-8702-14a1f7492d17" containerName="main" containerID="cri-o://68670bf50f362d7d4d8563da7a6fb236426e2ab81cd5c011751859a045ee6080" gracePeriod=45 Oct 30 16:56:33.828967 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.828934 6400 generic.go:356] "Generic (PLEG): container finished" podID="2530a183-f3eb-4326-8702-14a1f7492d17" containerID="68670bf50f362d7d4d8563da7a6fb236426e2ab81cd5c011751859a045ee6080" exitCode=0 Oct 30 16:56:33.829133 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.829002 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" event={"ID":"2530a183-f3eb-4326-8702-14a1f7492d17","Type":"ContainerDied","Data":"68670bf50f362d7d4d8563da7a6fb236426e2ab81cd5c011751859a045ee6080"} Oct 30 16:56:33.857198 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.857172 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:56:33.922783 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.922742 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-dshm\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-dshm\") pod \"2530a183-f3eb-4326-8702-14a1f7492d17\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " Oct 30 16:56:33.922975 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.922886 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2530a183-f3eb-4326-8702-14a1f7492d17-tls-certs\" (UniqueName: \"kubernetes.io/secret/2530a183-f3eb-4326-8702-14a1f7492d17-tls-certs\") pod \"2530a183-f3eb-4326-8702-14a1f7492d17\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " Oct 30 16:56:33.923048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.923030 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-home\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-home\") pod \"2530a183-f3eb-4326-8702-14a1f7492d17\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " Oct 30 16:56:33.923109 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.923096 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2530a183-f3eb-4326-8702-14a1f7492d17-kube-api-access-9r2wz\" (UniqueName: \"kubernetes.io/projected/2530a183-f3eb-4326-8702-14a1f7492d17-kube-api-access-9r2wz\") pod \"2530a183-f3eb-4326-8702-14a1f7492d17\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " Oct 30 16:56:33.923237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.923220 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-model-cache\") pod \"2530a183-f3eb-4326-8702-14a1f7492d17\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " Oct 30 16:56:33.923303 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.923277 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-home" pod "2530a183-f3eb-4326-8702-14a1f7492d17" (UID: "2530a183-f3eb-4326-8702-14a1f7492d17"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:33.923357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.923303 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-tmp-dir\") pod \"2530a183-f3eb-4326-8702-14a1f7492d17\" (UID: \"2530a183-f3eb-4326-8702-14a1f7492d17\") " Oct 30 16:56:33.923586 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.923515 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-tmp-dir" pod "2530a183-f3eb-4326-8702-14a1f7492d17" (UID: "2530a183-f3eb-4326-8702-14a1f7492d17"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:33.924408 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.924379 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:56:33.924512 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.924412 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:56:33.924692 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.924669 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-model-cache" pod "2530a183-f3eb-4326-8702-14a1f7492d17" (UID: "2530a183-f3eb-4326-8702-14a1f7492d17"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:33.925742 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.925717 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2530a183-f3eb-4326-8702-14a1f7492d17-tls-certs" pod "2530a183-f3eb-4326-8702-14a1f7492d17" (UID: "2530a183-f3eb-4326-8702-14a1f7492d17"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:33.925858 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.925795 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-dshm" pod "2530a183-f3eb-4326-8702-14a1f7492d17" (UID: "2530a183-f3eb-4326-8702-14a1f7492d17"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:56:33.926071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:33.926050 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2530a183-f3eb-4326-8702-14a1f7492d17-kube-api-access-9r2wz" pod "2530a183-f3eb-4326-8702-14a1f7492d17" (UID: "2530a183-f3eb-4326-8702-14a1f7492d17"). InnerVolumeSpecName "kube-api-access-9r2wz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:34.025853 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:34.025819 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:56:34.025853 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:34.025846 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2530a183-f3eb-4326-8702-14a1f7492d17-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:56:34.025853 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:34.025855 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9r2wz\" (UniqueName: \"kubernetes.io/projected/2530a183-f3eb-4326-8702-14a1f7492d17-kube-api-access-9r2wz\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:56:34.025853 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:34.025865 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2530a183-f3eb-4326-8702-14a1f7492d17-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:56:34.838296 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:34.838269 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" Oct 30 16:56:34.838296 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:34.838280 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt" event={"ID":"2530a183-f3eb-4326-8702-14a1f7492d17","Type":"ContainerDied","Data":"a50005abade8371d4f55f316a3d64cfad7c6a6a78f446367b71b15cebb4e72b2"} Oct 30 16:56:34.838866 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:34.838330 6400 scope.go:122] "RemoveContainer" containerID="68670bf50f362d7d4d8563da7a6fb236426e2ab81cd5c011751859a045ee6080" Oct 30 16:56:34.889203 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:34.889166 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt"] Oct 30 16:56:34.900528 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:34.900493 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-480a60/test-model-480a60-kserve-5794d9c9b5-twtrt"] Oct 30 16:56:34.969425 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:34.969394 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2530a183-f3eb-4326-8702-14a1f7492d17" path="/var/lib/kubelet/pods/2530a183-f3eb-4326-8702-14a1f7492d17/volumes" Oct 30 16:56:37.357741 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:37.357695 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z"] Oct 30 16:56:37.358222 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:37.357969 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" podUID="46fba546-bf94-4362-9b30-16d0aa688d8b" containerName="maas-api" containerID="cri-o://a8aee8bc11f5c18e2018e7d9eb1458920be90b3595c41bd44635ea69bd3bef48" gracePeriod=30 Oct 30 16:56:38.037642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.033921 6400 generic.go:356] "Generic (PLEG): container finished" podID="46fba546-bf94-4362-9b30-16d0aa688d8b" containerID="a8aee8bc11f5c18e2018e7d9eb1458920be90b3595c41bd44635ea69bd3bef48" exitCode=0 Oct 30 16:56:38.037642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.034134 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" event={"ID":"46fba546-bf94-4362-9b30-16d0aa688d8b","Type":"ContainerDied","Data":"a8aee8bc11f5c18e2018e7d9eb1458920be90b3595c41bd44635ea69bd3bef48"} Oct 30 16:56:38.037642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.034185 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" event={"ID":"46fba546-bf94-4362-9b30-16d0aa688d8b","Type":"ContainerDied","Data":"2cc270c9d1a3110661376705346c94f5d53b24c4e08905c94806c3aeaa6f8630"} Oct 30 16:56:38.037642 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.034206 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2cc270c9d1a3110661376705346c94f5d53b24c4e08905c94806c3aeaa6f8630" Oct 30 16:56:38.049921 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.049886 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:56:38.100562 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.100529 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/46fba546-bf94-4362-9b30-16d0aa688d8b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fba546-bf94-4362-9b30-16d0aa688d8b-openshift-service-ca\") pod \"46fba546-bf94-4362-9b30-16d0aa688d8b\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " Oct 30 16:56:38.100778 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.100673 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-maas-api-tls\") pod \"46fba546-bf94-4362-9b30-16d0aa688d8b\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " Oct 30 16:56:38.100967 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.100950 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/46fba546-bf94-4362-9b30-16d0aa688d8b-kube-api-access-t4g96\" (UniqueName: \"kubernetes.io/projected/46fba546-bf94-4362-9b30-16d0aa688d8b-kube-api-access-t4g96\") pod \"46fba546-bf94-4362-9b30-16d0aa688d8b\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " Oct 30 16:56:38.101137 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.101123 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-metrics-certs\") pod \"46fba546-bf94-4362-9b30-16d0aa688d8b\" (UID: \"46fba546-bf94-4362-9b30-16d0aa688d8b\") " Oct 30 16:56:38.101212 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.101151 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/46fba546-bf94-4362-9b30-16d0aa688d8b-openshift-service-ca" pod "46fba546-bf94-4362-9b30-16d0aa688d8b" (UID: "46fba546-bf94-4362-9b30-16d0aa688d8b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:56:38.101719 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.101698 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/46fba546-bf94-4362-9b30-16d0aa688d8b-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:56:38.103553 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.103529 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-metrics-certs" pod "46fba546-bf94-4362-9b30-16d0aa688d8b" (UID: "46fba546-bf94-4362-9b30-16d0aa688d8b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:38.103874 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.103856 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46fba546-bf94-4362-9b30-16d0aa688d8b-kube-api-access-t4g96" pod "46fba546-bf94-4362-9b30-16d0aa688d8b" (UID: "46fba546-bf94-4362-9b30-16d0aa688d8b"). InnerVolumeSpecName "kube-api-access-t4g96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:56:38.106384 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.106364 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-maas-api-tls" pod "46fba546-bf94-4362-9b30-16d0aa688d8b" (UID: "46fba546-bf94-4362-9b30-16d0aa688d8b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:56:38.203907 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.203684 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4g96\" (UniqueName: \"kubernetes.io/projected/46fba546-bf94-4362-9b30-16d0aa688d8b-kube-api-access-t4g96\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:56:38.203907 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.203719 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:56:38.203907 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:38.203733 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/46fba546-bf94-4362-9b30-16d0aa688d8b-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:56:39.042352 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:39.042314 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z" Oct 30 16:56:39.095575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:39.095540 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z"] Oct 30 16:56:39.106285 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:39.106256 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-ff1189-868f789f76-q9k8z"] Oct 30 16:56:40.969651 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:56:40.969592 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="46fba546-bf94-4362-9b30-16d0aa688d8b" path="/var/lib/kubelet/pods/46fba546-bf94-4362-9b30-16d0aa688d8b/volumes" Oct 30 16:56:54.606575 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:56:54.606528 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:57:22.458266 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.458230 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7"] Oct 30 16:57:22.458823 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.458441 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" podUID="422f5c36-cc16-4683-ad5c-af1a16c04714" containerName="discovery" containerID="cri-o://a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272" gracePeriod=30 Oct 30 16:57:22.539449 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.539412 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl"] Oct 30 16:57:22.539732 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.539701 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerName="istio-proxy" containerID="cri-o://f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978" gracePeriod=30 Oct 30 16:57:22.601116 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.600249 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp"] Oct 30 16:57:22.603206 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.602085 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" containerID="cri-o://04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323" gracePeriod=30 Oct 30 16:57:22.853393 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.853358 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:57:22.911507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.911197 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-kubeconfig\") pod \"422f5c36-cc16-4683-ad5c-af1a16c04714\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " Oct 30 16:57:22.911507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.911325 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-kube-api-access-tqjvl\" (UniqueName: \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-kube-api-access-tqjvl\") pod \"422f5c36-cc16-4683-ad5c-af1a16c04714\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " Oct 30 16:57:22.911507 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.911424 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-ca-configmap\") pod \"422f5c36-cc16-4683-ad5c-af1a16c04714\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " Oct 30 16:57:22.911856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.911527 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-dns-cert\") pod \"422f5c36-cc16-4683-ad5c-af1a16c04714\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " Oct 30 16:57:22.915532 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.915181 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-kube-api-access-tqjvl" pod "422f5c36-cc16-4683-ad5c-af1a16c04714" (UID: "422f5c36-cc16-4683-ad5c-af1a16c04714"). InnerVolumeSpecName "kube-api-access-tqjvl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:22.915919 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.915874 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-ca-configmap" pod "422f5c36-cc16-4683-ad5c-af1a16c04714" (UID: "422f5c36-cc16-4683-ad5c-af1a16c04714"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:22.917583 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.917522 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-kubeconfig" pod "422f5c36-cc16-4683-ad5c-af1a16c04714" (UID: "422f5c36-cc16-4683-ad5c-af1a16c04714"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:57:22.918558 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.918526 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-dns-cert" pod "422f5c36-cc16-4683-ad5c-af1a16c04714" (UID: "422f5c36-cc16-4683-ad5c-af1a16c04714"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:57:22.925391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.924234 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/422f5c36-cc16-4683-ad5c-af1a16c04714-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/422f5c36-cc16-4683-ad5c-af1a16c04714-local-certs\") pod \"422f5c36-cc16-4683-ad5c-af1a16c04714\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " Oct 30 16:57:22.925391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.924330 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-token\" (UniqueName: \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-token\") pod \"422f5c36-cc16-4683-ad5c-af1a16c04714\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " Oct 30 16:57:22.925391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.924363 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-cacerts\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-cacerts\") pod \"422f5c36-cc16-4683-ad5c-af1a16c04714\" (UID: \"422f5c36-cc16-4683-ad5c-af1a16c04714\") " Oct 30 16:57:22.925391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.925269 6400 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-kubeconfig\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:22.925391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.925293 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tqjvl\" (UniqueName: \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-kube-api-access-tqjvl\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:22.925391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.925307 6400 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-ca-configmap\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:22.925391 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.925321 6400 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-csr-dns-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:22.932081 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.932035 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/422f5c36-cc16-4683-ad5c-af1a16c04714-local-certs" pod "422f5c36-cc16-4683-ad5c-af1a16c04714" (UID: "422f5c36-cc16-4683-ad5c-af1a16c04714"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:22.932341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.932317 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-cacerts" pod "422f5c36-cc16-4683-ad5c-af1a16c04714" (UID: "422f5c36-cc16-4683-ad5c-af1a16c04714"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:57:22.939829 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:22.939796 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-token" pod "422f5c36-cc16-4683-ad5c-af1a16c04714" (UID: "422f5c36-cc16-4683-ad5c-af1a16c04714"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:23.027258 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.027224 6400 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/422f5c36-cc16-4683-ad5c-af1a16c04714-local-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:23.027447 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.027358 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/422f5c36-cc16-4683-ad5c-af1a16c04714-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:23.027447 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.027376 6400 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/422f5c36-cc16-4683-ad5c-af1a16c04714-cacerts\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:23.479192 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.479155 6400 generic.go:356] "Generic (PLEG): container finished" podID="422f5c36-cc16-4683-ad5c-af1a16c04714" containerID="a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272" exitCode=0 Oct 30 16:57:23.479705 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.479203 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" Oct 30 16:57:23.479705 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.479241 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" event={"ID":"422f5c36-cc16-4683-ad5c-af1a16c04714","Type":"ContainerDied","Data":"a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272"} Oct 30 16:57:23.479705 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.479313 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" event={"ID":"422f5c36-cc16-4683-ad5c-af1a16c04714","Type":"ContainerDied","Data":"9f1517241f415cca08c6c61e5b2951e9829eb6abbe2c7dad59bf1bb081eb29a6"} Oct 30 16:57:23.479705 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.479341 6400 scope.go:122] "RemoveContainer" containerID="a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272" Oct 30 16:57:23.494475 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.494454 6400 scope.go:122] "RemoveContainer" containerID="a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272" Oct 30 16:57:23.494911 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:57:23.494886 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272\": container with ID starting with a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272 not found: ID does not exist" containerID="a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272" Oct 30 16:57:23.495030 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.494924 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272"} err="failed to get container status \"a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272\": rpc error: code = NotFound desc = could not find container \"a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272\": container with ID starting with a59eabd7f348afe20d353c529bb3f4de789ba5db268281073738d64383745272 not found: ID does not exist" Oct 30 16:57:23.531213 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.531180 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7"] Oct 30 16:57:23.542575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:23.542542 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7"] Oct 30 16:57:24.606085 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:57:24.606042 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:57:24.971469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:24.971384 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="422f5c36-cc16-4683-ad5c-af1a16c04714" path="/var/lib/kubelet/pods/422f5c36-cc16-4683-ad5c-af1a16c04714/volumes" Oct 30 16:57:27.725783 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:27.725747 6400 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-7fhj7 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.105:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 16:57:27.726343 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:27.725830 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-7fhj7" podUID="422f5c36-cc16-4683-ad5c-af1a16c04714" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.105:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 16:57:27.935691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:27.935664 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:57:27.972480 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:27.972455 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:57:28.009451 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.009420 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-credential-socket\") pod \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " Oct 30 16:57:28.009666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.009486 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-envoy\") pod \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " Oct 30 16:57:28.009666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.009545 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istiod-ca-cert\") pod \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " Oct 30 16:57:28.009666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.009590 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-podinfo\") pod \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " Oct 30 16:57:28.009666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.009651 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-data\") pod \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " Oct 30 16:57:28.009900 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.009726 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-token\" (UniqueName: \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-token\") pod \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " Oct 30 16:57:28.009900 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.009779 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-credential-socket" pod "e4b1aabb-56f0-4e96-a71b-a272a57b99a3" (UID: "e4b1aabb-56f0-4e96-a71b-a272a57b99a3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:28.009900 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.009861 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-certs\") pod \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " Oct 30 16:57:28.010063 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.009967 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-kube-api-access-gmr4h\" (UniqueName: \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-kube-api-access-gmr4h\") pod \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " Oct 30 16:57:28.010063 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.010017 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-socket\") pod \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\" (UID: \"e4b1aabb-56f0-4e96-a71b-a272a57b99a3\") " Oct 30 16:57:28.010063 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.010034 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-data" pod "e4b1aabb-56f0-4e96-a71b-a272a57b99a3" (UID: "e4b1aabb-56f0-4e96-a71b-a272a57b99a3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:28.010263 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.010234 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-certs" pod "e4b1aabb-56f0-4e96-a71b-a272a57b99a3" (UID: "e4b1aabb-56f0-4e96-a71b-a272a57b99a3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:28.010934 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.010906 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.011038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.010936 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.011038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.010950 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.011158 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.011140 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-socket" pod "e4b1aabb-56f0-4e96-a71b-a272a57b99a3" (UID: "e4b1aabb-56f0-4e96-a71b-a272a57b99a3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:28.011453 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.011431 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istiod-ca-cert" pod "e4b1aabb-56f0-4e96-a71b-a272a57b99a3" (UID: "e4b1aabb-56f0-4e96-a71b-a272a57b99a3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:28.012962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.012940 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-kube-api-access-gmr4h" pod "e4b1aabb-56f0-4e96-a71b-a272a57b99a3" (UID: "e4b1aabb-56f0-4e96-a71b-a272a57b99a3"). InnerVolumeSpecName "kube-api-access-gmr4h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:28.013071 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.012974 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-token" pod "e4b1aabb-56f0-4e96-a71b-a272a57b99a3" (UID: "e4b1aabb-56f0-4e96-a71b-a272a57b99a3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:28.013852 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.013832 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-podinfo" pod "e4b1aabb-56f0-4e96-a71b-a272a57b99a3" (UID: "e4b1aabb-56f0-4e96-a71b-a272a57b99a3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:57:28.014524 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.014507 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-envoy" pod "e4b1aabb-56f0-4e96-a71b-a272a57b99a3" (UID: "e4b1aabb-56f0-4e96-a71b-a272a57b99a3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:28.112097 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112063 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-kube-api-access-m89p9\" (UniqueName: \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-kube-api-access-m89p9\") pod \"e7295aef-3d34-43db-9a22-71f3b9a6b554\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " Oct 30 16:57:28.112295 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112119 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-socket\") pod \"e7295aef-3d34-43db-9a22-71f3b9a6b554\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " Oct 30 16:57:28.112295 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112173 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-credential-socket\") pod \"e7295aef-3d34-43db-9a22-71f3b9a6b554\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " Oct 30 16:57:28.112295 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112237 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-token\" (UniqueName: \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-token\") pod \"e7295aef-3d34-43db-9a22-71f3b9a6b554\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " Oct 30 16:57:28.112295 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112270 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-data\") pod \"e7295aef-3d34-43db-9a22-71f3b9a6b554\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " Oct 30 16:57:28.112546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112344 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7295aef-3d34-43db-9a22-71f3b9a6b554-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7295aef-3d34-43db-9a22-71f3b9a6b554-istiod-ca-cert\") pod \"e7295aef-3d34-43db-9a22-71f3b9a6b554\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " Oct 30 16:57:28.112546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112401 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-certs\") pod \"e7295aef-3d34-43db-9a22-71f3b9a6b554\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " Oct 30 16:57:28.112546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112428 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-envoy\") pod \"e7295aef-3d34-43db-9a22-71f3b9a6b554\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " Oct 30 16:57:28.112546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112469 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-socket" pod "e7295aef-3d34-43db-9a22-71f3b9a6b554" (UID: "e7295aef-3d34-43db-9a22-71f3b9a6b554"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:28.112546 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112523 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-podinfo\") pod \"e7295aef-3d34-43db-9a22-71f3b9a6b554\" (UID: \"e7295aef-3d34-43db-9a22-71f3b9a6b554\") " Oct 30 16:57:28.112828 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112614 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-data" pod "e7295aef-3d34-43db-9a22-71f3b9a6b554" (UID: "e7295aef-3d34-43db-9a22-71f3b9a6b554"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:28.112828 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.112729 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-certs" pod "e7295aef-3d34-43db-9a22-71f3b9a6b554" (UID: "e7295aef-3d34-43db-9a22-71f3b9a6b554"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:28.113107 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113013 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7295aef-3d34-43db-9a22-71f3b9a6b554-istiod-ca-cert" pod "e7295aef-3d34-43db-9a22-71f3b9a6b554" (UID: "e7295aef-3d34-43db-9a22-71f3b9a6b554"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:28.113409 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113379 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-credential-socket" pod "e7295aef-3d34-43db-9a22-71f3b9a6b554" (UID: "e7295aef-3d34-43db-9a22-71f3b9a6b554"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:28.113641 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113622 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.113890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113646 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7295aef-3d34-43db-9a22-71f3b9a6b554-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.113890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113661 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmr4h\" (UniqueName: \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-kube-api-access-gmr4h\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.113890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113679 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.113890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113693 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.113890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113707 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.113890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113720 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.113890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113735 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.113890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113749 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e4b1aabb-56f0-4e96-a71b-a272a57b99a3-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.113890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113761 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.113890 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.113775 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.114715 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.114693 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-token" pod "e7295aef-3d34-43db-9a22-71f3b9a6b554" (UID: "e7295aef-3d34-43db-9a22-71f3b9a6b554"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:28.114822 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.114783 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-kube-api-access-m89p9" pod "e7295aef-3d34-43db-9a22-71f3b9a6b554" (UID: "e7295aef-3d34-43db-9a22-71f3b9a6b554"). InnerVolumeSpecName "kube-api-access-m89p9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:28.115025 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.115000 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-podinfo" pod "e7295aef-3d34-43db-9a22-71f3b9a6b554" (UID: "e7295aef-3d34-43db-9a22-71f3b9a6b554"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:57:28.115353 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.115336 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-envoy" pod "e7295aef-3d34-43db-9a22-71f3b9a6b554" (UID: "e7295aef-3d34-43db-9a22-71f3b9a6b554"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:57:28.215249 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.215166 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m89p9\" (UniqueName: \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-kube-api-access-m89p9\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.215249 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.215196 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.215249 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.215205 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.215249 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.215215 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7295aef-3d34-43db-9a22-71f3b9a6b554-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:28.525049 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.525004 6400 generic.go:356] "Generic (PLEG): container finished" podID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerID="f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978" exitCode=0 Oct 30 16:57:28.525252 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.525071 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" Oct 30 16:57:28.525252 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.525099 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" event={"ID":"e4b1aabb-56f0-4e96-a71b-a272a57b99a3","Type":"ContainerDied","Data":"f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978"} Oct 30 16:57:28.525252 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.525163 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl" event={"ID":"e4b1aabb-56f0-4e96-a71b-a272a57b99a3","Type":"ContainerDied","Data":"c95d98c635cbe3da20aee95f4678d9d3074db95819f96d2103b0f9dbf827e022"} Oct 30 16:57:28.525252 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.525198 6400 scope.go:122] "RemoveContainer" containerID="f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978" Oct 30 16:57:28.526841 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.526818 6400 generic.go:356] "Generic (PLEG): container finished" podID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerID="04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323" exitCode=0 Oct 30 16:57:28.526962 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.526859 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" Oct 30 16:57:28.527015 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.526951 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" event={"ID":"e7295aef-3d34-43db-9a22-71f3b9a6b554","Type":"ContainerDied","Data":"04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323"} Oct 30 16:57:28.527015 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.527001 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp" event={"ID":"e7295aef-3d34-43db-9a22-71f3b9a6b554","Type":"ContainerDied","Data":"cef8254979bcddbd2bbdfdb49880f5fa001c6125854e76f2dee90d043bc950ba"} Oct 30 16:57:28.538291 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.538271 6400 scope.go:122] "RemoveContainer" containerID="f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978" Oct 30 16:57:28.538557 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:57:28.538537 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978\": container with ID starting with f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978 not found: ID does not exist" containerID="f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978" Oct 30 16:57:28.538674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.538565 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978"} err="failed to get container status \"f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978\": rpc error: code = NotFound desc = could not find container \"f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978\": container with ID starting with f5d7a30c5bc45eaffb30bbcabcbb195c98403ab7d7255e01205d05a72f4a5978 not found: ID does not exist" Oct 30 16:57:28.538674 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.538582 6400 scope.go:122] "RemoveContainer" containerID="04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323" Oct 30 16:57:28.548975 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.548955 6400 scope.go:122] "RemoveContainer" containerID="04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323" Oct 30 16:57:28.549269 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:57:28.549244 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323\": container with ID starting with 04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323 not found: ID does not exist" containerID="04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323" Oct 30 16:57:28.549332 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.549280 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323"} err="failed to get container status \"04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323\": rpc error: code = NotFound desc = could not find container \"04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323\": container with ID starting with 04df6050e03c3e01dbc31453cb7030134e280526d5599181367b04a973706323 not found: ID does not exist" Oct 30 16:57:28.593966 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.593937 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp"] Oct 30 16:57:28.610989 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.610954 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-nwvdp"] Oct 30 16:57:28.633043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.633015 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl"] Oct 30 16:57:28.638643 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.638618 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-x7tpl"] Oct 30 16:57:28.969567 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.969483 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" path="/var/lib/kubelet/pods/e4b1aabb-56f0-4e96-a71b-a272a57b99a3/volumes" Oct 30 16:57:28.970100 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:28.970083 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" path="/var/lib/kubelet/pods/e7295aef-3d34-43db-9a22-71f3b9a6b554/volumes" Oct 30 16:57:30.510267 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:30.510231 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7"] Oct 30 16:57:30.510819 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:30.510484 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" podUID="e6b96ac8-4863-48fc-80bc-d93b140a3095" containerName="maas-api" containerID="cri-o://d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd" gracePeriod=30 Oct 30 16:57:31.084108 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.083787 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:57:31.269680 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.269643 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-maas-api-tls\") pod \"e6b96ac8-4863-48fc-80bc-d93b140a3095\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " Oct 30 16:57:31.269850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.269787 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b96ac8-4863-48fc-80bc-d93b140a3095-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b96ac8-4863-48fc-80bc-d93b140a3095-openshift-service-ca\") pod \"e6b96ac8-4863-48fc-80bc-d93b140a3095\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " Oct 30 16:57:31.269850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.269845 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b96ac8-4863-48fc-80bc-d93b140a3095-kube-api-access-7jsq9\" (UniqueName: \"kubernetes.io/projected/e6b96ac8-4863-48fc-80bc-d93b140a3095-kube-api-access-7jsq9\") pod \"e6b96ac8-4863-48fc-80bc-d93b140a3095\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " Oct 30 16:57:31.270009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.269982 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-metrics-certs\") pod \"e6b96ac8-4863-48fc-80bc-d93b140a3095\" (UID: \"e6b96ac8-4863-48fc-80bc-d93b140a3095\") " Oct 30 16:57:31.270275 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.270243 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b96ac8-4863-48fc-80bc-d93b140a3095-openshift-service-ca" pod "e6b96ac8-4863-48fc-80bc-d93b140a3095" (UID: "e6b96ac8-4863-48fc-80bc-d93b140a3095"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:57:31.270887 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.270862 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e6b96ac8-4863-48fc-80bc-d93b140a3095-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:31.272401 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.272367 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b96ac8-4863-48fc-80bc-d93b140a3095-kube-api-access-7jsq9" pod "e6b96ac8-4863-48fc-80bc-d93b140a3095" (UID: "e6b96ac8-4863-48fc-80bc-d93b140a3095"). InnerVolumeSpecName "kube-api-access-7jsq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:57:31.272814 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.272792 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-maas-api-tls" pod "e6b96ac8-4863-48fc-80bc-d93b140a3095" (UID: "e6b96ac8-4863-48fc-80bc-d93b140a3095"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:57:31.272935 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.272845 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-metrics-certs" pod "e6b96ac8-4863-48fc-80bc-d93b140a3095" (UID: "e6b96ac8-4863-48fc-80bc-d93b140a3095"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:57:31.372993 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.372957 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:31.372993 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.372987 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jsq9\" (UniqueName: \"kubernetes.io/projected/e6b96ac8-4863-48fc-80bc-d93b140a3095-kube-api-access-7jsq9\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:31.372993 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.372997 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e6b96ac8-4863-48fc-80bc-d93b140a3095-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:57:31.560088 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.560001 6400 generic.go:356] "Generic (PLEG): container finished" podID="e6b96ac8-4863-48fc-80bc-d93b140a3095" containerID="d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd" exitCode=0 Oct 30 16:57:31.560088 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.560058 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" Oct 30 16:57:31.560693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.560085 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" event={"ID":"e6b96ac8-4863-48fc-80bc-d93b140a3095","Type":"ContainerDied","Data":"d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd"} Oct 30 16:57:31.560693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.560146 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7" event={"ID":"e6b96ac8-4863-48fc-80bc-d93b140a3095","Type":"ContainerDied","Data":"9be378c785fa9f9c4c1086de18338c33f2b0538d64a2ee1b73afaa35d11602dd"} Oct 30 16:57:31.560693 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.560174 6400 scope.go:122] "RemoveContainer" containerID="d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd" Oct 30 16:57:31.574202 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.574176 6400 scope.go:122] "RemoveContainer" containerID="d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd" Oct 30 16:57:31.574536 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:57:31.574503 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd\": container with ID starting with d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd not found: ID does not exist" containerID="d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd" Oct 30 16:57:31.574666 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.574543 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd"} err="failed to get container status \"d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd\": rpc error: code = NotFound desc = could not find container \"d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd\": container with ID starting with d2a72af59a0a9899a375334aa51f30256e329736389777c9a697c32cf936e6cd not found: ID does not exist" Oct 30 16:57:31.617262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.617232 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7"] Oct 30 16:57:31.627873 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.627845 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-480a60-754d96fd5f-r4lq7"] Oct 30 16:57:31.631135 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:31.631098 6400 scope.go:122] "RemoveContainer" containerID="a8aee8bc11f5c18e2018e7d9eb1458920be90b3595c41bd44635ea69bd3bef48" Oct 30 16:57:32.978258 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:32.978218 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b96ac8-4863-48fc-80bc-d93b140a3095" path="/var/lib/kubelet/pods/e6b96ac8-4863-48fc-80bc-d93b140a3095/volumes" Oct 30 16:57:33.658986 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.658952 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p"] Oct 30 16:57:33.660127 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660096 6400 cpu_manager.go:414] "containerMap: removing container" podUID="46fba546-bf94-4362-9b30-16d0aa688d8b" containerName="maas-api" Oct 30 16:57:33.660127 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660126 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="46fba546-bf94-4362-9b30-16d0aa688d8b" containerName="maas-api" Oct 30 16:57:33.660298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660143 6400 cpu_manager.go:414] "containerMap: removing container" podUID="422f5c36-cc16-4683-ad5c-af1a16c04714" containerName="discovery" Oct 30 16:57:33.660298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660150 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="422f5c36-cc16-4683-ad5c-af1a16c04714" containerName="discovery" Oct 30 16:57:33.660298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660205 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" Oct 30 16:57:33.660298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660213 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" Oct 30 16:57:33.660298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660225 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerName="istio-proxy" Oct 30 16:57:33.660298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660233 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerName="istio-proxy" Oct 30 16:57:33.660298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660255 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e6b96ac8-4863-48fc-80bc-d93b140a3095" containerName="maas-api" Oct 30 16:57:33.660298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660262 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b96ac8-4863-48fc-80bc-d93b140a3095" containerName="maas-api" Oct 30 16:57:33.660298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660283 6400 cpu_manager.go:414] "containerMap: removing container" podUID="2530a183-f3eb-4326-8702-14a1f7492d17" containerName="main" Oct 30 16:57:33.660298 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660290 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2530a183-f3eb-4326-8702-14a1f7492d17" containerName="main" Oct 30 16:57:33.660762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660453 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7295aef-3d34-43db-9a22-71f3b9a6b554" containerName="istio-proxy" Oct 30 16:57:33.660762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660465 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4b1aabb-56f0-4e96-a71b-a272a57b99a3" containerName="istio-proxy" Oct 30 16:57:33.660762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660472 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b96ac8-4863-48fc-80bc-d93b140a3095" containerName="maas-api" Oct 30 16:57:33.660762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660480 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="2530a183-f3eb-4326-8702-14a1f7492d17" containerName="main" Oct 30 16:57:33.660762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660491 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="46fba546-bf94-4362-9b30-16d0aa688d8b" containerName="maas-api" Oct 30 16:57:33.660762 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.660498 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="422f5c36-cc16-4683-ad5c-af1a16c04714" containerName="discovery" Oct 30 16:57:33.666707 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.666683 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.696495 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.696466 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p"] Oct 30 16:57:33.708573 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.708532 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.708790 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.708666 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38e5e4e-341a-458f-93f1-771f464ddb8d-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.708790 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.708713 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.708790 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.708771 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.709013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.708876 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.709013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.708908 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.709013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.708951 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.709013 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.708986 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cr9n\" (UniqueName: \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-kube-api-access-7cr9n\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.709217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.709055 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.810972 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.810936 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811130 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.810990 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cr9n\" (UniqueName: \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-kube-api-access-7cr9n\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811130 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811052 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811171 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811224 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811210 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38e5e4e-341a-458f-93f1-771f464ddb8d-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811304 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811244 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811357 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811311 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811415 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811388 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811469 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811450 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-credential-socket\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811529 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811465 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811582 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811530 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811698 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811666 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-data\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.811873 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.811852 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-certs\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.812307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.812278 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38e5e4e-341a-458f-93f1-771f464ddb8d-istiod-ca-cert\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.813768 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.813742 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-envoy\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.813981 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.813964 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-podinfo\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.837647 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.837616 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-token\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.837832 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.837784 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cr9n\" (UniqueName: \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-kube-api-access-7cr9n\") pod \"e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:33.983101 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:33.983072 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:34.307818 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:57:34.307791 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38e5e4e_341a_458f_93f1_771f464ddb8d.slice/crio-c70612df2dee9c1a8afe28107ab2481296f767b877d7df8de8b2af3599db1805 WatchSource:0}: Error finding container c70612df2dee9c1a8afe28107ab2481296f767b877d7df8de8b2af3599db1805: Status 404 returned error can't find the container with id c70612df2dee9c1a8afe28107ab2481296f767b877d7df8de8b2af3599db1805 Oct 30 16:57:34.309251 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:34.309229 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p"] Oct 30 16:57:34.310914 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:34.310880 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:57:34.311023 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:34.310948 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:57:34.311023 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:34.310977 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 16:57:34.586731 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:34.586586 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" event={"ID":"b38e5e4e-341a-458f-93f1-771f464ddb8d","Type":"ContainerStarted","Data":"f4f6cf3d9bf600fc393f65f4e7c1db637b9c9e88784d9db31945d9ca63770c40"} Oct 30 16:57:34.586731 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:34.586711 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" event={"ID":"b38e5e4e-341a-458f-93f1-771f464ddb8d","Type":"ContainerStarted","Data":"c70612df2dee9c1a8afe28107ab2481296f767b877d7df8de8b2af3599db1805"} Oct 30 16:57:34.633497 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:34.633440 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" podStartSLOduration=1.633426694 podStartE2EDuration="1.633426694s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:34.629400132 +0000 UTC m=+2163.779681120" watchObservedRunningTime="2026-10-30 16:57:34.633426694 +0000 UTC m=+2163.783707681" Oct 30 16:57:34.983961 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:34.983937 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:35.985938 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:35.985899 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:57:35.986410 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:35.985964 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 16:57:36.983757 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:36.983725 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:57:36.983911 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:36.983779 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 16:57:37.983965 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:37.983932 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:57:37.984386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:37.983986 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 16:57:38.984207 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:38.984167 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:57:38.984662 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:38.984227 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 16:57:39.984420 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:39.984384 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 16:57:39.984863 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:39.984439 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 16:57:41.135521 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:41.134483 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:41.135521 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:41.135104 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:41.136737 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:41.136714 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 16:57:51.789018 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.788981 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h"] Oct 30 16:57:51.794510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.794485 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:51.814930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.814894 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h"] Oct 30 16:57:51.822653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.820852 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-dc74d5\"/\"openshift-service-ca.crt\"" Oct 30 16:57:51.822653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.820854 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-dc74d5\"/\"auth-test-model-dc74d5-kserve-self-signed-certs\"" Oct 30 16:57:51.822653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.820982 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-dc74d5\"/\"kube-root-ca.crt\"" Oct 30 16:57:51.822653 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.822324 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-dc74d5\"/\"default-dockercfg-lthgr\"" Oct 30 16:57:51.923944 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.923901 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ls928\" (UniqueName: \"kubernetes.io/projected/28e0f55b-622b-4ecf-bb99-5b080c872f4c-kube-api-access-ls928\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:51.924116 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.924055 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-home\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:51.924205 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.924162 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-dshm\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:51.924358 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.924336 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tls-certs\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:51.924420 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.924392 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-model-cache\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:51.924575 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:51.924556 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tmp-dir\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.026058 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.026011 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ls928\" (UniqueName: \"kubernetes.io/projected/28e0f55b-622b-4ecf-bb99-5b080c872f4c-kube-api-access-ls928\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.026268 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.026127 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-home\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.026340 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.026290 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-dshm\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.026480 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.026459 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tls-certs\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.026550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.026506 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-home\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.026550 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.026515 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-model-cache\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.026690 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.026678 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tmp-dir\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.026835 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.026810 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-model-cache\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.027114 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.027092 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tmp-dir\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.029178 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.029159 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tls-certs\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.029266 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.029175 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-dshm\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.054547 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.054465 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ls928\" (UniqueName: \"kubernetes.io/projected/28e0f55b-622b-4ecf-bb99-5b080c872f4c-kube-api-access-ls928\") pod \"auth-test-model-dc74d5-kserve-565d7557c-jf44h\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.124622 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.122894 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:52.514937 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.514910 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h"] Oct 30 16:57:52.515281 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:57:52.515255 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28e0f55b_622b_4ecf_bb99_5b080c872f4c.slice/crio-2be94f6a72f6c7da296b212e49e592ff586eaa9d1e734d4354d3f36f8b324460 WatchSource:0}: Error finding container 2be94f6a72f6c7da296b212e49e592ff586eaa9d1e734d4354d3f36f8b324460: Status 404 returned error can't find the container with id 2be94f6a72f6c7da296b212e49e592ff586eaa9d1e734d4354d3f36f8b324460 Oct 30 16:57:52.770084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.769995 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" event={"ID":"28e0f55b-622b-4ecf-bb99-5b080c872f4c","Type":"ContainerStarted","Data":"7d7e25a31d355788ca4ba1858ed81fcb5b6e29b5b74c8b524229e439d9facf37"} Oct 30 16:57:52.770084 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.770044 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" event={"ID":"28e0f55b-622b-4ecf-bb99-5b080c872f4c","Type":"ContainerStarted","Data":"2be94f6a72f6c7da296b212e49e592ff586eaa9d1e734d4354d3f36f8b324460"} Oct 30 16:57:52.816254 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:52.816189 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" podStartSLOduration=1.816170214 podStartE2EDuration="1.816170214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:57:52.811150761 +0000 UTC m=+2181.961431749" watchObservedRunningTime="2026-10-30 16:57:52.816170214 +0000 UTC m=+2181.966451204" Oct 30 16:57:53.124281 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:57:53.124200 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:57:54.603968 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:57:54.603924 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:58:02.124363 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:02.124329 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:58:02.137687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:02.137658 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:58:02.858392 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:02.858364 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 16:58:03.842494 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:03.842456 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g"] Oct 30 16:58:03.846840 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:03.846822 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:03.867981 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:03.867949 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w0-bfecdd-serving-cert\"" Oct 30 16:58:03.878618 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:03.878573 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g"] Oct 30 16:58:03.920848 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:03.920793 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:03.921043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:03.920948 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:03.921314 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:03.921286 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:03.921449 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:03.921335 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xqwwg\" (UniqueName: \"kubernetes.io/projected/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-kube-api-access-xqwwg\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:04.022622 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.022567 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:04.022829 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.022670 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xqwwg\" (UniqueName: \"kubernetes.io/projected/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-kube-api-access-xqwwg\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:04.022829 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.022782 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:04.022953 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.022850 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:04.023390 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.023363 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:04.025502 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.025484 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-maas-api-tls\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:04.025559 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.025510 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-metrics-certs\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:04.047349 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.047316 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xqwwg\" (UniqueName: \"kubernetes.io/projected/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-kube-api-access-xqwwg\") pod \"maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:04.167544 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.167446 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:04.487922 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:58:04.487893 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb97e4c4_3a72_4ee0_b98b_4b8e53107bea.slice/crio-46f8bb52fd653b4a3eab91f7b9b197cd41d61d55a97a84436a3af8b0fa5b0813 WatchSource:0}: Error finding container 46f8bb52fd653b4a3eab91f7b9b197cd41d61d55a97a84436a3af8b0fa5b0813: Status 404 returned error can't find the container with id 46f8bb52fd653b4a3eab91f7b9b197cd41d61d55a97a84436a3af8b0fa5b0813 Oct 30 16:58:04.488703 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.488681 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g"] Oct 30 16:58:04.489837 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.489775 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 16:58:04.866971 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:04.866875 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" event={"ID":"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea","Type":"ContainerStarted","Data":"46f8bb52fd653b4a3eab91f7b9b197cd41d61d55a97a84436a3af8b0fa5b0813"} Oct 30 16:58:05.875994 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:05.875951 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" event={"ID":"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea","Type":"ContainerStarted","Data":"0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089"} Oct 30 16:58:05.876427 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:05.876026 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:05.918586 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:05.918532 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" podStartSLOduration=2.497342429 podStartE2EDuration="2.918516342s" totalImagesPullingTime="421.173913ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:05.917165442 +0000 UTC m=+2195.067446432" watchObservedRunningTime="2026-10-30 16:58:05.918516342 +0000 UTC m=+2195.068797329" Oct 30 16:58:11.888371 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:11.888342 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 16:58:23.207708 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:23.207668 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv"] Oct 30 16:58:24.602023 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:58:24.601986 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:58:38.255244 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.255160 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" podUID="bda2dd6e-eacb-429c-aac2-982f3d1e8ade" containerName="main" containerID="cri-o://9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd" gracePeriod=45 Oct 30 16:58:38.571724 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.571698 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:58:38.739192 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.739155 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-model-cache\") pod \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " Oct 30 16:58:38.739386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.739218 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-kube-api-access-2fhcv\" (UniqueName: \"kubernetes.io/projected/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-kube-api-access-2fhcv\") pod \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " Oct 30 16:58:38.739386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.739260 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tls-certs\" (UniqueName: \"kubernetes.io/secret/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tls-certs\") pod \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " Oct 30 16:58:38.739386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.739286 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-home\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-home\") pod \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " Oct 30 16:58:38.739386 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.739305 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tmp-dir\") pod \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " Oct 30 16:58:38.739647 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.739470 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-model-cache" pod "bda2dd6e-eacb-429c-aac2-982f3d1e8ade" (UID: "bda2dd6e-eacb-429c-aac2-982f3d1e8ade"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:38.739647 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.739488 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-dshm\") pod \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\" (UID: \"bda2dd6e-eacb-429c-aac2-982f3d1e8ade\") " Oct 30 16:58:38.739769 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.739640 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-home" pod "bda2dd6e-eacb-429c-aac2-982f3d1e8ade" (UID: "bda2dd6e-eacb-429c-aac2-982f3d1e8ade"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:38.739769 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.739651 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tmp-dir" pod "bda2dd6e-eacb-429c-aac2-982f3d1e8ade" (UID: "bda2dd6e-eacb-429c-aac2-982f3d1e8ade"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:38.740724 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.740694 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:58:38.740724 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.740725 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:58:38.740930 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.740741 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:58:38.741905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.741881 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-dshm" pod "bda2dd6e-eacb-429c-aac2-982f3d1e8ade" (UID: "bda2dd6e-eacb-429c-aac2-982f3d1e8ade"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:58:38.742009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.741909 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tls-certs" pod "bda2dd6e-eacb-429c-aac2-982f3d1e8ade" (UID: "bda2dd6e-eacb-429c-aac2-982f3d1e8ade"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:58:38.742009 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.741920 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-kube-api-access-2fhcv" pod "bda2dd6e-eacb-429c-aac2-982f3d1e8ade" (UID: "bda2dd6e-eacb-429c-aac2-982f3d1e8ade"). InnerVolumeSpecName "kube-api-access-2fhcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:58:38.842293 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.842216 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:58:38.842293 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.842251 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fhcv\" (UniqueName: \"kubernetes.io/projected/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-kube-api-access-2fhcv\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:58:38.842293 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:38.842262 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bda2dd6e-eacb-429c-aac2-982f3d1e8ade-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:58:39.138089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:39.137998 6400 generic.go:356] "Generic (PLEG): container finished" podID="bda2dd6e-eacb-429c-aac2-982f3d1e8ade" containerID="9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd" exitCode=0 Oct 30 16:58:39.138089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:39.138040 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" Oct 30 16:58:39.138089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:39.138073 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" event={"ID":"bda2dd6e-eacb-429c-aac2-982f3d1e8ade","Type":"ContainerDied","Data":"9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd"} Oct 30 16:58:39.138359 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:39.138119 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv" event={"ID":"bda2dd6e-eacb-429c-aac2-982f3d1e8ade","Type":"ContainerDied","Data":"47cd34f1669b2b29f465464512a15ebf3de039a01759fe19a5afc7c11b7db362"} Oct 30 16:58:39.138359 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:39.138140 6400 scope.go:122] "RemoveContainer" containerID="9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd" Oct 30 16:58:39.148849 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:39.148829 6400 scope.go:122] "RemoveContainer" containerID="9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd" Oct 30 16:58:39.149172 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:58:39.149152 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd\": container with ID starting with 9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd not found: ID does not exist" containerID="9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd" Oct 30 16:58:39.149249 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:39.149185 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd"} err="failed to get container status \"9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd\": rpc error: code = NotFound desc = could not find container \"9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd\": container with ID starting with 9aa9d3459aeaec286de273c47d15215fbfe6cb1416d39c5714ddedcdb690c5dd not found: ID does not exist" Oct 30 16:58:39.191765 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:39.191731 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv"] Oct 30 16:58:39.202911 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:39.202882 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-668047/test-model-668047-kserve-6f45d6bd77-j8zjv"] Oct 30 16:58:40.969159 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:40.969128 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bda2dd6e-eacb-429c-aac2-982f3d1e8ade" path="/var/lib/kubelet/pods/bda2dd6e-eacb-429c-aac2-982f3d1e8ade/volumes" Oct 30 16:58:53.745523 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.745490 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz"] Oct 30 16:58:53.746634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.746580 6400 cpu_manager.go:414] "containerMap: removing container" podUID="bda2dd6e-eacb-429c-aac2-982f3d1e8ade" containerName="main" Oct 30 16:58:53.746634 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.746618 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bda2dd6e-eacb-429c-aac2-982f3d1e8ade" containerName="main" Oct 30 16:58:53.746832 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.746817 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bda2dd6e-eacb-429c-aac2-982f3d1e8ade" containerName="main" Oct 30 16:58:53.750724 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.750701 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.776635 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.776583 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz"] Oct 30 16:58:53.779051 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.777805 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-661509\"/\"kube-root-ca.crt\"" Oct 30 16:58:53.779051 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.777832 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-661509\"/\"openshift-service-ca.crt\"" Oct 30 16:58:53.780425 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.780404 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-661509\"/\"default-dockercfg-z2f8h\"" Oct 30 16:58:53.780581 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.780549 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-661509\"/\"auth-test-model-661509-kserve-self-signed-certs\"" Oct 30 16:58:53.896312 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.896250 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9zdk\" (UniqueName: \"kubernetes.io/projected/81b60d6e-bef0-499e-9cb5-7c935fc0587e-kube-api-access-p9zdk\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.896312 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.896323 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-home\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.896557 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.896414 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tls-certs\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.896557 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.896530 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-model-cache\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.896681 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.896624 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-dshm\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.896735 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.896722 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tmp-dir\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.998711 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.998462 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9zdk\" (UniqueName: \"kubernetes.io/projected/81b60d6e-bef0-499e-9cb5-7c935fc0587e-kube-api-access-p9zdk\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.998711 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.998532 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-home\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.998711 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.998566 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tls-certs\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.998711 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.998686 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-model-cache\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.999074 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.998780 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-dshm\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:53.999074 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.998868 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tmp-dir\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:54.007657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:53.999465 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tmp-dir\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:54.007657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:54.000090 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-home\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:54.007657 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:54.002385 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-model-cache\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:54.023655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:54.021885 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tls-certs\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:54.023655 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:54.022333 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-dshm\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:54.030625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:54.025417 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9zdk\" (UniqueName: \"kubernetes.io/projected/81b60d6e-bef0-499e-9cb5-7c935fc0587e-kube-api-access-p9zdk\") pod \"auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:54.071316 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:54.071196 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:54.453082 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:58:54.453049 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod81b60d6e_bef0_499e_9cb5_7c935fc0587e.slice/crio-9871c33b0a7f9e71cff30c9848792f0a3786d839cae2f87809fec1af052eda0b WatchSource:0}: Error finding container 9871c33b0a7f9e71cff30c9848792f0a3786d839cae2f87809fec1af052eda0b: Status 404 returned error can't find the container with id 9871c33b0a7f9e71cff30c9848792f0a3786d839cae2f87809fec1af052eda0b Oct 30 16:58:54.454616 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:54.454571 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz"] Oct 30 16:58:54.602536 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:58:54.602394 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:58:55.283517 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:55.283484 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" event={"ID":"81b60d6e-bef0-499e-9cb5-7c935fc0587e","Type":"ContainerStarted","Data":"0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa"} Oct 30 16:58:55.283517 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:55.283527 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" event={"ID":"81b60d6e-bef0-499e-9cb5-7c935fc0587e","Type":"ContainerStarted","Data":"9871c33b0a7f9e71cff30c9848792f0a3786d839cae2f87809fec1af052eda0b"} Oct 30 16:58:55.336972 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:55.336672 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" podStartSLOduration=2.336652308 podStartE2EDuration="2.336652308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:58:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:58:55.33600125 +0000 UTC m=+2244.486282239" watchObservedRunningTime="2026-10-30 16:58:55.336652308 +0000 UTC m=+2244.486933297" Oct 30 16:58:56.072824 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:56.072792 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:58:59.157072 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:58:59.157031 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm"] Oct 30 16:59:04.072389 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:04.072347 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:59:04.086226 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:04.086201 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:59:04.430985 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:04.430901 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 16:59:05.938412 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:05.938376 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q"] Oct 30 16:59:05.938939 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:05.938577 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" containerID="cri-o://38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e" gracePeriod=30 Oct 30 16:59:05.949780 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:05.949750 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4"] Oct 30 16:59:05.950056 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:05.950029 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" containerID="cri-o://b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2" gracePeriod=30 Oct 30 16:59:07.350645 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:07.350562 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j"] Oct 30 16:59:07.351150 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:07.350842 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" containerID="cri-o://220dff0781bdf872793c1ddfeb32a95e6d0a472c715bea50838860a4b108ffad" gracePeriod=30 Oct 30 16:59:07.611936 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:07.611831 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h"] Oct 30 16:59:07.612155 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:07.612127 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" containerID="cri-o://6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd" gracePeriod=30 Oct 30 16:59:10.583038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:10.583002 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg"] Oct 30 16:59:10.583808 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:10.583260 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" containerID="cri-o://ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93" gracePeriod=30 Oct 30 16:59:11.369899 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.369867 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:59:11.402659 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.402594 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:59:11.494024 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.493986 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-data\") pod \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " Oct 30 16:59:11.494237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494043 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-credential-socket\") pod \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " Oct 30 16:59:11.494237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494084 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-socket\") pod \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " Oct 30 16:59:11.494237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494116 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-certs\") pod \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " Oct 30 16:59:11.494237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494180 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-token\" (UniqueName: \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-token\") pod \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " Oct 30 16:59:11.494237 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494213 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istiod-ca-cert\") pod \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " Oct 30 16:59:11.494547 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494308 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-envoy\") pod \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " Oct 30 16:59:11.494547 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494327 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-credential-socket" pod "605b7c33-6b21-4aa8-bfa4-c85298b8b717" (UID: "605b7c33-6b21-4aa8-bfa4-c85298b8b717"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:11.494547 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494498 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-podinfo\") pod \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " Oct 30 16:59:11.494782 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494568 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-certs" pod "605b7c33-6b21-4aa8-bfa4-c85298b8b717" (UID: "605b7c33-6b21-4aa8-bfa4-c85298b8b717"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:11.494782 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494655 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-kube-api-access-5lrl2\" (UniqueName: \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-kube-api-access-5lrl2\") pod \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\" (UID: \"605b7c33-6b21-4aa8-bfa4-c85298b8b717\") " Oct 30 16:59:11.494782 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.494755 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-socket" pod "605b7c33-6b21-4aa8-bfa4-c85298b8b717" (UID: "605b7c33-6b21-4aa8-bfa4-c85298b8b717"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:11.495432 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.495398 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istiod-ca-cert" pod "605b7c33-6b21-4aa8-bfa4-c85298b8b717" (UID: "605b7c33-6b21-4aa8-bfa4-c85298b8b717"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:11.495559 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.495514 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.495559 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.495535 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.495559 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.495553 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.495783 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.495567 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.496872 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.496840 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-data" pod "605b7c33-6b21-4aa8-bfa4-c85298b8b717" (UID: "605b7c33-6b21-4aa8-bfa4-c85298b8b717"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:11.498293 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.498226 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-token" pod "605b7c33-6b21-4aa8-bfa4-c85298b8b717" (UID: "605b7c33-6b21-4aa8-bfa4-c85298b8b717"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:11.499126 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.499105 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-envoy" pod "605b7c33-6b21-4aa8-bfa4-c85298b8b717" (UID: "605b7c33-6b21-4aa8-bfa4-c85298b8b717"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:11.499506 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.499453 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-podinfo" pod "605b7c33-6b21-4aa8-bfa4-c85298b8b717" (UID: "605b7c33-6b21-4aa8-bfa4-c85298b8b717"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:11.500819 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.500793 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-kube-api-access-5lrl2" pod "605b7c33-6b21-4aa8-bfa4-c85298b8b717" (UID: "605b7c33-6b21-4aa8-bfa4-c85298b8b717"). InnerVolumeSpecName "kube-api-access-5lrl2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:11.503184 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.503158 6400 generic.go:356] "Generic (PLEG): container finished" podID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerID="38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e" exitCode=0 Oct 30 16:59:11.503274 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.503191 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" event={"ID":"628cf0ef-f9e0-4469-a4aa-5528721b5818","Type":"ContainerDied","Data":"38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e"} Oct 30 16:59:11.503274 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.503199 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" Oct 30 16:59:11.503274 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.503244 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q" event={"ID":"628cf0ef-f9e0-4469-a4aa-5528721b5818","Type":"ContainerDied","Data":"b9fa02f6a44c5804359a1fa84513e924f209d695d0de3dd8fba3a36ee04bcafc"} Oct 30 16:59:11.503274 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.503272 6400 scope.go:122] "RemoveContainer" containerID="38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e" Oct 30 16:59:11.504883 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.504849 6400 generic.go:356] "Generic (PLEG): container finished" podID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerID="b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2" exitCode=0 Oct 30 16:59:11.504990 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.504896 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" Oct 30 16:59:11.504990 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.504936 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" event={"ID":"605b7c33-6b21-4aa8-bfa4-c85298b8b717","Type":"ContainerDied","Data":"b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2"} Oct 30 16:59:11.504990 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.504979 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4" event={"ID":"605b7c33-6b21-4aa8-bfa4-c85298b8b717","Type":"ContainerDied","Data":"f8e232c3d0f8de38bf46a146535086eb3d0a799e89a7701d20dfb77bd882b83c"} Oct 30 16:59:11.516860 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.516844 6400 scope.go:122] "RemoveContainer" containerID="38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e" Oct 30 16:59:11.517145 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:11.517125 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e\": container with ID starting with 38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e not found: ID does not exist" containerID="38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e" Oct 30 16:59:11.517214 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.517157 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e"} err="failed to get container status \"38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e\": rpc error: code = NotFound desc = could not find container \"38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e\": container with ID starting with 38563d8ca654deb8f2bfe7e9e64e68d04ec8e54ae2723c609d6e127567d8036e not found: ID does not exist" Oct 30 16:59:11.517214 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.517176 6400 scope.go:122] "RemoveContainer" containerID="b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2" Oct 30 16:59:11.528089 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.528066 6400 scope.go:122] "RemoveContainer" containerID="b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2" Oct 30 16:59:11.528384 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:11.528364 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2\": container with ID starting with b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2 not found: ID does not exist" containerID="b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2" Oct 30 16:59:11.528442 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.528392 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2"} err="failed to get container status \"b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2\": rpc error: code = NotFound desc = could not find container \"b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2\": container with ID starting with b7865d8de40cb86a850d3a58ca4890c3e246b690cfd497d1a013deabd95103d2 not found: ID does not exist" Oct 30 16:59:11.557342 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.557303 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4"] Oct 30 16:59:11.575043 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.575007 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4cwj4"] Oct 30 16:59:11.596200 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.596164 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-socket\") pod \"628cf0ef-f9e0-4469-a4aa-5528721b5818\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " Oct 30 16:59:11.596816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.596324 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/628cf0ef-f9e0-4469-a4aa-5528721b5818-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/628cf0ef-f9e0-4469-a4aa-5528721b5818-istiod-ca-cert\") pod \"628cf0ef-f9e0-4469-a4aa-5528721b5818\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " Oct 30 16:59:11.596816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.596390 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-token\" (UniqueName: \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-token\") pod \"628cf0ef-f9e0-4469-a4aa-5528721b5818\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " Oct 30 16:59:11.596816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.596418 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-credential-socket\") pod \"628cf0ef-f9e0-4469-a4aa-5528721b5818\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " Oct 30 16:59:11.596816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.596481 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-podinfo\") pod \"628cf0ef-f9e0-4469-a4aa-5528721b5818\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " Oct 30 16:59:11.596816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.596629 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-data\") pod \"628cf0ef-f9e0-4469-a4aa-5528721b5818\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " Oct 30 16:59:11.596816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.596663 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-certs\") pod \"628cf0ef-f9e0-4469-a4aa-5528721b5818\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " Oct 30 16:59:11.596816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.596703 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-kube-api-access-hcljr\" (UniqueName: \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-kube-api-access-hcljr\") pod \"628cf0ef-f9e0-4469-a4aa-5528721b5818\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " Oct 30 16:59:11.596816 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.596732 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-envoy\") pod \"628cf0ef-f9e0-4469-a4aa-5528721b5818\" (UID: \"628cf0ef-f9e0-4469-a4aa-5528721b5818\") " Oct 30 16:59:11.600670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.597569 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.600670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.597617 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.600670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.597633 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5lrl2\" (UniqueName: \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-kube-api-access-5lrl2\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.600670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.597647 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.600670 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.597660 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/605b7c33-6b21-4aa8-bfa4-c85298b8b717-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.606307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.602783 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-credential-socket" pod "628cf0ef-f9e0-4469-a4aa-5528721b5818" (UID: "628cf0ef-f9e0-4469-a4aa-5528721b5818"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:11.606307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.602916 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-data" pod "628cf0ef-f9e0-4469-a4aa-5528721b5818" (UID: "628cf0ef-f9e0-4469-a4aa-5528721b5818"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:11.606307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.603230 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-certs" pod "628cf0ef-f9e0-4469-a4aa-5528721b5818" (UID: "628cf0ef-f9e0-4469-a4aa-5528721b5818"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:11.606307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.603442 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-socket" pod "628cf0ef-f9e0-4469-a4aa-5528721b5818" (UID: "628cf0ef-f9e0-4469-a4aa-5528721b5818"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:11.606307 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.603861 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/628cf0ef-f9e0-4469-a4aa-5528721b5818-istiod-ca-cert" pod "628cf0ef-f9e0-4469-a4aa-5528721b5818" (UID: "628cf0ef-f9e0-4469-a4aa-5528721b5818"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:11.614629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.608698 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-envoy" pod "628cf0ef-f9e0-4469-a4aa-5528721b5818" (UID: "628cf0ef-f9e0-4469-a4aa-5528721b5818"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:11.614629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.609181 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-podinfo" pod "628cf0ef-f9e0-4469-a4aa-5528721b5818" (UID: "628cf0ef-f9e0-4469-a4aa-5528721b5818"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:11.614629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.613395 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-token" pod "628cf0ef-f9e0-4469-a4aa-5528721b5818" (UID: "628cf0ef-f9e0-4469-a4aa-5528721b5818"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:11.614629 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.614567 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-kube-api-access-hcljr" pod "628cf0ef-f9e0-4469-a4aa-5528721b5818" (UID: "628cf0ef-f9e0-4469-a4aa-5528721b5818"). InnerVolumeSpecName "kube-api-access-hcljr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:11.698856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.698816 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.698856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.698846 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/628cf0ef-f9e0-4469-a4aa-5528721b5818-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.698856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.698856 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.699104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.698869 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.699104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.698886 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.699104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.698894 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.699104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.698902 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.699104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.698910 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hcljr\" (UniqueName: \"kubernetes.io/projected/628cf0ef-f9e0-4469-a4aa-5528721b5818-kube-api-access-hcljr\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.699104 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.698918 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/628cf0ef-f9e0-4469-a4aa-5528721b5818-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:11.850182 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.850140 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q"] Oct 30 16:59:11.856530 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:11.856498 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-jfk8q"] Oct 30 16:59:12.427333 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.427289 6400 patch_prober.go:29] interesting pod/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 16:59:12.427510 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.427371 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 16:59:12.516526 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.516489 6400 generic.go:356] "Generic (PLEG): container finished" podID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerID="220dff0781bdf872793c1ddfeb32a95e6d0a472c715bea50838860a4b108ffad" exitCode=0 Oct 30 16:59:12.516745 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.516558 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" event={"ID":"35adff5a-315a-4e33-922c-ec81f3091a8f","Type":"ContainerDied","Data":"220dff0781bdf872793c1ddfeb32a95e6d0a472c715bea50838860a4b108ffad"} Oct 30 16:59:12.725085 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.725016 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:59:12.819956 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.819687 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-certs\") pod \"35adff5a-315a-4e33-922c-ec81f3091a8f\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " Oct 30 16:59:12.819956 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.819749 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-envoy\") pod \"35adff5a-315a-4e33-922c-ec81f3091a8f\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " Oct 30 16:59:12.819956 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.819819 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-credential-socket\") pod \"35adff5a-315a-4e33-922c-ec81f3091a8f\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " Oct 30 16:59:12.820175 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.820069 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-credential-socket" pod "35adff5a-315a-4e33-922c-ec81f3091a8f" (UID: "35adff5a-315a-4e33-922c-ec81f3091a8f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:12.820782 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.820506 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:12.820782 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.820555 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-certs" pod "35adff5a-315a-4e33-922c-ec81f3091a8f" (UID: "35adff5a-315a-4e33-922c-ec81f3091a8f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:12.823210 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.823184 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-envoy" pod "35adff5a-315a-4e33-922c-ec81f3091a8f" (UID: "35adff5a-315a-4e33-922c-ec81f3091a8f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:12.921419 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.921386 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-podinfo\") pod \"35adff5a-315a-4e33-922c-ec81f3091a8f\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " Oct 30 16:59:12.921533 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.921508 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-kube-api-access-64rtl\" (UniqueName: \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-kube-api-access-64rtl\") pod \"35adff5a-315a-4e33-922c-ec81f3091a8f\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " Oct 30 16:59:12.921764 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.921747 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-socket\") pod \"35adff5a-315a-4e33-922c-ec81f3091a8f\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " Oct 30 16:59:12.921833 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.921784 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-data\") pod \"35adff5a-315a-4e33-922c-ec81f3091a8f\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " Oct 30 16:59:12.921833 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.921817 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-token\" (UniqueName: \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-token\") pod \"35adff5a-315a-4e33-922c-ec81f3091a8f\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " Oct 30 16:59:12.921942 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.921845 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35adff5a-315a-4e33-922c-ec81f3091a8f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35adff5a-315a-4e33-922c-ec81f3091a8f-istiod-ca-cert\") pod \"35adff5a-315a-4e33-922c-ec81f3091a8f\" (UID: \"35adff5a-315a-4e33-922c-ec81f3091a8f\") " Oct 30 16:59:12.922080 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.922044 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-socket" pod "35adff5a-315a-4e33-922c-ec81f3091a8f" (UID: "35adff5a-315a-4e33-922c-ec81f3091a8f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:12.922752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.922372 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-data" pod "35adff5a-315a-4e33-922c-ec81f3091a8f" (UID: "35adff5a-315a-4e33-922c-ec81f3091a8f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:12.922752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.922463 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35adff5a-315a-4e33-922c-ec81f3091a8f-istiod-ca-cert" pod "35adff5a-315a-4e33-922c-ec81f3091a8f" (UID: "35adff5a-315a-4e33-922c-ec81f3091a8f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:12.922752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.922644 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:12.922752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.922666 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:12.922752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.922681 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35adff5a-315a-4e33-922c-ec81f3091a8f-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:12.922752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.922693 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:12.922752 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.922706 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:12.923924 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.923898 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-kube-api-access-64rtl" pod "35adff5a-315a-4e33-922c-ec81f3091a8f" (UID: "35adff5a-315a-4e33-922c-ec81f3091a8f"). InnerVolumeSpecName "kube-api-access-64rtl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:12.924464 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.924439 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-podinfo" pod "35adff5a-315a-4e33-922c-ec81f3091a8f" (UID: "35adff5a-315a-4e33-922c-ec81f3091a8f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:12.924722 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.924702 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-token" pod "35adff5a-315a-4e33-922c-ec81f3091a8f" (UID: "35adff5a-315a-4e33-922c-ec81f3091a8f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:12.930848 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.930824 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:59:12.968648 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.968590 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" path="/var/lib/kubelet/pods/605b7c33-6b21-4aa8-bfa4-c85298b8b717/volumes" Oct 30 16:59:12.969209 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:12.969193 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" path="/var/lib/kubelet/pods/628cf0ef-f9e0-4469-a4aa-5528721b5818/volumes" Oct 30 16:59:13.023990 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.023946 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/745f06d2-beda-4d0f-8f95-322d0ea28329-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745f06d2-beda-4d0f-8f95-322d0ea28329-istiod-ca-cert\") pod \"745f06d2-beda-4d0f-8f95-322d0ea28329\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " Oct 30 16:59:13.024196 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.024021 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-socket\") pod \"745f06d2-beda-4d0f-8f95-322d0ea28329\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " Oct 30 16:59:13.024278 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.024218 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-token\" (UniqueName: \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-token\") pod \"745f06d2-beda-4d0f-8f95-322d0ea28329\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " Oct 30 16:59:13.024278 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.024252 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-kube-api-access-q2nt9\" (UniqueName: \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-kube-api-access-q2nt9\") pod \"745f06d2-beda-4d0f-8f95-322d0ea28329\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " Oct 30 16:59:13.024399 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.024287 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-credential-socket\") pod \"745f06d2-beda-4d0f-8f95-322d0ea28329\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " Oct 30 16:59:13.024399 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.024346 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-podinfo\") pod \"745f06d2-beda-4d0f-8f95-322d0ea28329\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " Oct 30 16:59:13.024399 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.024368 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/745f06d2-beda-4d0f-8f95-322d0ea28329-istiod-ca-cert" pod "745f06d2-beda-4d0f-8f95-322d0ea28329" (UID: "745f06d2-beda-4d0f-8f95-322d0ea28329"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:13.024399 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.024388 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-data\") pod \"745f06d2-beda-4d0f-8f95-322d0ea28329\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " Oct 30 16:59:13.024625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.024427 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-certs\") pod \"745f06d2-beda-4d0f-8f95-322d0ea28329\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " Oct 30 16:59:13.024625 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.024491 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-envoy\") pod \"745f06d2-beda-4d0f-8f95-322d0ea28329\" (UID: \"745f06d2-beda-4d0f-8f95-322d0ea28329\") " Oct 30 16:59:13.024882 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.024849 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-socket" pod "745f06d2-beda-4d0f-8f95-322d0ea28329" (UID: "745f06d2-beda-4d0f-8f95-322d0ea28329"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:13.026004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.025316 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-data" pod "745f06d2-beda-4d0f-8f95-322d0ea28329" (UID: "745f06d2-beda-4d0f-8f95-322d0ea28329"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:13.026004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.025397 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.026004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.025424 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64rtl\" (UniqueName: \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-kube-api-access-64rtl\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.026004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.025439 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/745f06d2-beda-4d0f-8f95-322d0ea28329-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.026004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.025453 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.026004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.025467 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35adff5a-315a-4e33-922c-ec81f3091a8f-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.026004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.025465 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-certs" pod "745f06d2-beda-4d0f-8f95-322d0ea28329" (UID: "745f06d2-beda-4d0f-8f95-322d0ea28329"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:13.026004 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.025960 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-credential-socket" pod "745f06d2-beda-4d0f-8f95-322d0ea28329" (UID: "745f06d2-beda-4d0f-8f95-322d0ea28329"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:13.028151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.028026 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-envoy" pod "745f06d2-beda-4d0f-8f95-322d0ea28329" (UID: "745f06d2-beda-4d0f-8f95-322d0ea28329"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:13.028151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.028567 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-token" pod "745f06d2-beda-4d0f-8f95-322d0ea28329" (UID: "745f06d2-beda-4d0f-8f95-322d0ea28329"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:13.028151 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.028575 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-kube-api-access-q2nt9" pod "745f06d2-beda-4d0f-8f95-322d0ea28329" (UID: "745f06d2-beda-4d0f-8f95-322d0ea28329"). InnerVolumeSpecName "kube-api-access-q2nt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:13.030218 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.030192 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-podinfo" pod "745f06d2-beda-4d0f-8f95-322d0ea28329" (UID: "745f06d2-beda-4d0f-8f95-322d0ea28329"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:13.127424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.127393 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.127424 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.127425 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2nt9\" (UniqueName: \"kubernetes.io/projected/745f06d2-beda-4d0f-8f95-322d0ea28329-kube-api-access-q2nt9\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.127687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.127439 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.127687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.127455 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.127687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.127463 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.127687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.127471 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.127687 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.127480 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/745f06d2-beda-4d0f-8f95-322d0ea28329-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:13.528219 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.528179 6400 generic.go:356] "Generic (PLEG): container finished" podID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerID="6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd" exitCode=0 Oct 30 16:59:13.528420 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.528225 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" Oct 30 16:59:13.528420 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.528253 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" event={"ID":"745f06d2-beda-4d0f-8f95-322d0ea28329","Type":"ContainerDied","Data":"6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd"} Oct 30 16:59:13.528420 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.528302 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h" event={"ID":"745f06d2-beda-4d0f-8f95-322d0ea28329","Type":"ContainerDied","Data":"9d7ca96d0f6687ded6d76e372e9be44efbcd5bf11653e584803b91916fcab0c3"} Oct 30 16:59:13.528420 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.528333 6400 scope.go:122] "RemoveContainer" containerID="6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd" Oct 30 16:59:13.530066 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.529985 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" Oct 30 16:59:13.530211 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.529987 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j" event={"ID":"35adff5a-315a-4e33-922c-ec81f3091a8f","Type":"ContainerDied","Data":"b5852fb2abaa346881f7493f14d4655571bd7beeb25f59158921692c5277bfe3"} Oct 30 16:59:13.540905 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.540887 6400 scope.go:122] "RemoveContainer" containerID="6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd" Oct 30 16:59:13.541145 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:13.541124 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd\": container with ID starting with 6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd not found: ID does not exist" containerID="6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd" Oct 30 16:59:13.541199 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.541154 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd"} err="failed to get container status \"6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd\": rpc error: code = NotFound desc = could not find container \"6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd\": container with ID starting with 6ec19711d6b8085028ccbcc9680b25282739a9c51ebac6ae696c018f5e1b06dd not found: ID does not exist" Oct 30 16:59:13.541199 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.541171 6400 scope.go:122] "RemoveContainer" containerID="220dff0781bdf872793c1ddfeb32a95e6d0a472c715bea50838860a4b108ffad" Oct 30 16:59:13.581867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.581791 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h"] Oct 30 16:59:13.592656 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.592582 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-tp58h"] Oct 30 16:59:13.623676 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.623646 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j"] Oct 30 16:59:13.627326 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:13.627296 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-750dcb-openshift-default-777f567b9f-hwv6j"] Oct 30 16:59:14.187265 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.187210 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" podUID="97637696-5890-4712-b677-f9ef36fdf467" containerName="main" containerID="cri-o://46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c" gracePeriod=45 Oct 30 16:59:14.198124 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.198084 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" podUID="97637696-5890-4712-b677-f9ef36fdf467" containerName="main" probeResult="failure" output="Get \"https://10.134.0.124:8000/ready\": dial tcp 10.134.0.124:8000: connect: connection refused" Oct 30 16:59:14.493656 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.493625 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:59:14.540970 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.540920 6400 generic.go:356] "Generic (PLEG): container finished" podID="97637696-5890-4712-b677-f9ef36fdf467" containerID="46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c" exitCode=0 Oct 30 16:59:14.541166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.540997 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" Oct 30 16:59:14.541166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.540988 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" event={"ID":"97637696-5890-4712-b677-f9ef36fdf467","Type":"ContainerDied","Data":"46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c"} Oct 30 16:59:14.541166 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.541152 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm" event={"ID":"97637696-5890-4712-b677-f9ef36fdf467","Type":"ContainerDied","Data":"96681b616564e44804ec54c4c65e37188f83bcc87027f8a6b19fc954b0e51639"} Oct 30 16:59:14.541341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.541172 6400 scope.go:122] "RemoveContainer" containerID="46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c" Oct 30 16:59:14.555633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.553635 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-model-cache\") pod \"97637696-5890-4712-b677-f9ef36fdf467\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " Oct 30 16:59:14.555633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.553920 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-home\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-home\") pod \"97637696-5890-4712-b677-f9ef36fdf467\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " Oct 30 16:59:14.555633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.553957 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-dshm\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-dshm\") pod \"97637696-5890-4712-b677-f9ef36fdf467\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " Oct 30 16:59:14.555633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.554055 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-tmp-dir\") pod \"97637696-5890-4712-b677-f9ef36fdf467\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " Oct 30 16:59:14.555633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.554126 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-home" pod "97637696-5890-4712-b677-f9ef36fdf467" (UID: "97637696-5890-4712-b677-f9ef36fdf467"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:14.555633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.554167 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97637696-5890-4712-b677-f9ef36fdf467-kube-api-access-6jtqg\" (UniqueName: \"kubernetes.io/projected/97637696-5890-4712-b677-f9ef36fdf467-kube-api-access-6jtqg\") pod \"97637696-5890-4712-b677-f9ef36fdf467\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " Oct 30 16:59:14.555633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.554202 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97637696-5890-4712-b677-f9ef36fdf467-tls-certs\" (UniqueName: \"kubernetes.io/secret/97637696-5890-4712-b677-f9ef36fdf467-tls-certs\") pod \"97637696-5890-4712-b677-f9ef36fdf467\" (UID: \"97637696-5890-4712-b677-f9ef36fdf467\") " Oct 30 16:59:14.555633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.555247 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:14.555633 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.555618 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-model-cache" pod "97637696-5890-4712-b677-f9ef36fdf467" (UID: "97637696-5890-4712-b677-f9ef36fdf467"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:14.556218 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.555851 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-tmp-dir" pod "97637696-5890-4712-b677-f9ef36fdf467" (UID: "97637696-5890-4712-b677-f9ef36fdf467"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:14.558341 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.558308 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-dshm" pod "97637696-5890-4712-b677-f9ef36fdf467" (UID: "97637696-5890-4712-b677-f9ef36fdf467"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:14.561824 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.561742 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97637696-5890-4712-b677-f9ef36fdf467-tls-certs" pod "97637696-5890-4712-b677-f9ef36fdf467" (UID: "97637696-5890-4712-b677-f9ef36fdf467"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:14.561824 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.561788 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97637696-5890-4712-b677-f9ef36fdf467-kube-api-access-6jtqg" pod "97637696-5890-4712-b677-f9ef36fdf467" (UID: "97637696-5890-4712-b677-f9ef36fdf467"). InnerVolumeSpecName "kube-api-access-6jtqg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:14.562865 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.562760 6400 scope.go:122] "RemoveContainer" containerID="46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c" Oct 30 16:59:14.563189 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:14.563158 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c\": container with ID starting with 46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c not found: ID does not exist" containerID="46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c" Oct 30 16:59:14.563280 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.563201 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c"} err="failed to get container status \"46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c\": rpc error: code = NotFound desc = could not find container \"46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c\": container with ID starting with 46b92c810c33ef7ebd210d7cb1d299bf35a4c7b6f220982dd75ca1810d4a9b2c not found: ID does not exist" Oct 30 16:59:14.656527 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.656491 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:14.656740 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.656547 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:14.656740 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.656561 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jtqg\" (UniqueName: \"kubernetes.io/projected/97637696-5890-4712-b677-f9ef36fdf467-kube-api-access-6jtqg\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:14.656740 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.656571 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/97637696-5890-4712-b677-f9ef36fdf467-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:14.656740 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.656581 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/97637696-5890-4712-b677-f9ef36fdf467-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:14.888511 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.888478 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm"] Oct 30 16:59:14.899531 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.899506 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-9bfcac/test-model-9bfcac-kserve-fb56cbb95-276pm"] Oct 30 16:59:14.969737 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.969699 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" path="/var/lib/kubelet/pods/35adff5a-315a-4e33-922c-ec81f3091a8f/volumes" Oct 30 16:59:14.970308 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.970287 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" path="/var/lib/kubelet/pods/745f06d2-beda-4d0f-8f95-322d0ea28329/volumes" Oct 30 16:59:14.971078 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:14.971060 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97637696-5890-4712-b677-f9ef36fdf467" path="/var/lib/kubelet/pods/97637696-5890-4712-b677-f9ef36fdf467/volumes" Oct 30 16:59:15.853947 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.853885 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:59:15.974327 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974298 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-istio-token\" (UniqueName: \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-istio-token\") pod \"e7623e9c-0154-4059-bf52-027533652d9e\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " Oct 30 16:59:15.974587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974374 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-data\") pod \"e7623e9c-0154-4059-bf52-027533652d9e\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " Oct 30 16:59:15.974587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974403 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7623e9c-0154-4059-bf52-027533652d9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7623e9c-0154-4059-bf52-027533652d9e-istiod-ca-cert\") pod \"e7623e9c-0154-4059-bf52-027533652d9e\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " Oct 30 16:59:15.974587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974459 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-credential-socket\") pod \"e7623e9c-0154-4059-bf52-027533652d9e\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " Oct 30 16:59:15.974587 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974499 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-certs\") pod \"e7623e9c-0154-4059-bf52-027533652d9e\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " Oct 30 16:59:15.974867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974672 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-envoy\") pod \"e7623e9c-0154-4059-bf52-027533652d9e\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " Oct 30 16:59:15.974867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974732 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-kube-api-access-vr94s\" (UniqueName: \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-kube-api-access-vr94s\") pod \"e7623e9c-0154-4059-bf52-027533652d9e\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " Oct 30 16:59:15.974867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974739 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-data" pod "e7623e9c-0154-4059-bf52-027533652d9e" (UID: "e7623e9c-0154-4059-bf52-027533652d9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:15.974867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974798 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-credential-socket" pod "e7623e9c-0154-4059-bf52-027533652d9e" (UID: "e7623e9c-0154-4059-bf52-027533652d9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:15.974867 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974802 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-socket\") pod \"e7623e9c-0154-4059-bf52-027533652d9e\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " Oct 30 16:59:15.975119 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974894 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-certs" pod "e7623e9c-0154-4059-bf52-027533652d9e" (UID: "e7623e9c-0154-4059-bf52-027533652d9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:15.975119 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974906 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7623e9c-0154-4059-bf52-027533652d9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7623e9c-0154-4059-bf52-027533652d9e-istio-podinfo\") pod \"e7623e9c-0154-4059-bf52-027533652d9e\" (UID: \"e7623e9c-0154-4059-bf52-027533652d9e\") " Oct 30 16:59:15.975119 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.974907 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7623e9c-0154-4059-bf52-027533652d9e-istiod-ca-cert" pod "e7623e9c-0154-4059-bf52-027533652d9e" (UID: "e7623e9c-0154-4059-bf52-027533652d9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:15.975119 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.975046 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-socket" pod "e7623e9c-0154-4059-bf52-027533652d9e" (UID: "e7623e9c-0154-4059-bf52-027533652d9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:15.975856 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.975831 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.975968 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.975861 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7623e9c-0154-4059-bf52-027533652d9e-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.975968 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.975897 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.975968 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.975914 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.975968 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.975929 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:15.976835 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.976803 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-istio-token" pod "e7623e9c-0154-4059-bf52-027533652d9e" (UID: "e7623e9c-0154-4059-bf52-027533652d9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:15.977062 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.977039 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-envoy" pod "e7623e9c-0154-4059-bf52-027533652d9e" (UID: "e7623e9c-0154-4059-bf52-027533652d9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 16:59:15.977165 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.977107 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-kube-api-access-vr94s" pod "e7623e9c-0154-4059-bf52-027533652d9e" (UID: "e7623e9c-0154-4059-bf52-027533652d9e"). InnerVolumeSpecName "kube-api-access-vr94s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:15.977280 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:15.977261 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7623e9c-0154-4059-bf52-027533652d9e-istio-podinfo" pod "e7623e9c-0154-4059-bf52-027533652d9e" (UID: "e7623e9c-0154-4059-bf52-027533652d9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 16:59:16.077246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.077218 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7623e9c-0154-4059-bf52-027533652d9e-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:16.077246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.077243 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr94s\" (UniqueName: \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-kube-api-access-vr94s\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:16.077246 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.077253 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7623e9c-0154-4059-bf52-027533652d9e-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:16.077482 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.077261 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7623e9c-0154-4059-bf52-027533652d9e-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:16.560789 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.560757 6400 generic.go:356] "Generic (PLEG): container finished" podID="e7623e9c-0154-4059-bf52-027533652d9e" containerID="ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93" exitCode=0 Oct 30 16:59:16.560974 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.560806 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" Oct 30 16:59:16.560974 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.560835 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" event={"ID":"e7623e9c-0154-4059-bf52-027533652d9e","Type":"ContainerDied","Data":"ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93"} Oct 30 16:59:16.560974 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.560901 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg" event={"ID":"e7623e9c-0154-4059-bf52-027533652d9e","Type":"ContainerDied","Data":"f0d8f18625e7e8acef03b0d869a942e98e43a2410ef9b0e72473e48800dad00c"} Oct 30 16:59:16.560974 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.560930 6400 scope.go:122] "RemoveContainer" containerID="ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93" Oct 30 16:59:16.572445 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.572425 6400 scope.go:122] "RemoveContainer" containerID="40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47" Oct 30 16:59:16.582987 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.582962 6400 scope.go:122] "RemoveContainer" containerID="ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93" Oct 30 16:59:16.583261 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:16.583243 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93\": container with ID starting with ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93 not found: ID does not exist" containerID="ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93" Oct 30 16:59:16.583349 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.583270 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93"} err="failed to get container status \"ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93\": rpc error: code = NotFound desc = could not find container \"ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93\": container with ID starting with ec5d41f5b86059f14bb8cfd713848b1e5ecc785740a9c0a0ab740b6870767b93 not found: ID does not exist" Oct 30 16:59:16.583349 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.583289 6400 scope.go:122] "RemoveContainer" containerID="40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47" Oct 30 16:59:16.583531 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:16.583511 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47\": container with ID starting with 40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47 not found: ID does not exist" containerID="40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47" Oct 30 16:59:16.583590 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.583540 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47"} err="failed to get container status \"40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47\": rpc error: code = NotFound desc = could not find container \"40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47\": container with ID starting with 40750e24123037ec318b9c757a725a7101a193d2f136adac7c5be04929e1af47 not found: ID does not exist" Oct 30 16:59:16.623763 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.623740 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg"] Oct 30 16:59:16.634832 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.634811 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-1b7bb3-openshift-default-584475499c-skhtg"] Oct 30 16:59:16.968731 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:16.968652 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7623e9c-0154-4059-bf52-027533652d9e" path="/var/lib/kubelet/pods/e7623e9c-0154-4059-bf52-027533652d9e/volumes" Oct 30 16:59:24.600992 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:24.600957 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 16:59:44.065925 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.065875 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr"] Oct 30 16:59:44.066703 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.066511 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" podUID="68b35351-443b-40e5-b5a4-f548366bba8e" containerName="maas-api" containerID="cri-o://ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492" gracePeriod=30 Oct 30 16:59:44.511899 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.511853 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:59:44.522048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.521183 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/68b35351-443b-40e5-b5a4-f548366bba8e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b35351-443b-40e5-b5a4-f548366bba8e-openshift-service-ca\") pod \"68b35351-443b-40e5-b5a4-f548366bba8e\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " Oct 30 16:59:44.522048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.521372 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/68b35351-443b-40e5-b5a4-f548366bba8e-kube-api-access-bfwzk\" (UniqueName: \"kubernetes.io/projected/68b35351-443b-40e5-b5a4-f548366bba8e-kube-api-access-bfwzk\") pod \"68b35351-443b-40e5-b5a4-f548366bba8e\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " Oct 30 16:59:44.522048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.521558 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-metrics-certs\") pod \"68b35351-443b-40e5-b5a4-f548366bba8e\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " Oct 30 16:59:44.522048 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.521641 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-maas-api-tls\") pod \"68b35351-443b-40e5-b5a4-f548366bba8e\" (UID: \"68b35351-443b-40e5-b5a4-f548366bba8e\") " Oct 30 16:59:44.524850 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.524817 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68b35351-443b-40e5-b5a4-f548366bba8e-openshift-service-ca" pod "68b35351-443b-40e5-b5a4-f548366bba8e" (UID: "68b35351-443b-40e5-b5a4-f548366bba8e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:44.531647 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.531537 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/68b35351-443b-40e5-b5a4-f548366bba8e-kube-api-access-bfwzk" pod "68b35351-443b-40e5-b5a4-f548366bba8e" (UID: "68b35351-443b-40e5-b5a4-f548366bba8e"). InnerVolumeSpecName "kube-api-access-bfwzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:44.531801 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.531742 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-maas-api-tls" pod "68b35351-443b-40e5-b5a4-f548366bba8e" (UID: "68b35351-443b-40e5-b5a4-f548366bba8e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:44.541163 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.541121 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-metrics-certs" pod "68b35351-443b-40e5-b5a4-f548366bba8e" (UID: "68b35351-443b-40e5-b5a4-f548366bba8e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:44.623697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.623658 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/68b35351-443b-40e5-b5a4-f548366bba8e-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:44.623697 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.623698 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfwzk\" (UniqueName: \"kubernetes.io/projected/68b35351-443b-40e5-b5a4-f548366bba8e-kube-api-access-bfwzk\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:44.623950 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.623713 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:44.623950 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.623727 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/68b35351-443b-40e5-b5a4-f548366bba8e-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:44.787027 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.786994 6400 generic.go:356] "Generic (PLEG): container finished" podID="68b35351-443b-40e5-b5a4-f548366bba8e" containerID="ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492" exitCode=0 Oct 30 16:59:44.787217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.787037 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" Oct 30 16:59:44.787217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.787073 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" event={"ID":"68b35351-443b-40e5-b5a4-f548366bba8e","Type":"ContainerDied","Data":"ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492"} Oct 30 16:59:44.787217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.787129 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr" event={"ID":"68b35351-443b-40e5-b5a4-f548366bba8e","Type":"ContainerDied","Data":"b57590b287620275402de0e9b28af581073694f5aa3065884681a7ee84762a9d"} Oct 30 16:59:44.787217 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.787154 6400 scope.go:122] "RemoveContainer" containerID="ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492" Oct 30 16:59:44.797986 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.797962 6400 scope.go:122] "RemoveContainer" containerID="ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492" Oct 30 16:59:44.798291 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:44.798269 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492\": container with ID starting with ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492 not found: ID does not exist" containerID="ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492" Oct 30 16:59:44.798382 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.798303 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492"} err="failed to get container status \"ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492\": rpc error: code = NotFound desc = could not find container \"ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492\": container with ID starting with ffebb1a07a044154839be34aec198d34f6868b94f9c0c0cd37f1f884f1ba8492 not found: ID does not exist" Oct 30 16:59:44.857597 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.857554 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr"] Oct 30 16:59:44.876230 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.876200 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-a59502-7f8db578dc-cgggr"] Oct 30 16:59:44.973622 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.973574 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="68b35351-443b-40e5-b5a4-f548366bba8e" path="/var/lib/kubelet/pods/68b35351-443b-40e5-b5a4-f548366bba8e/volumes" Oct 30 16:59:44.983559 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.983524 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb"] Oct 30 16:59:44.985262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985233 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" Oct 30 16:59:44.985262 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985262 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985284 6400 cpu_manager.go:414] "containerMap: removing container" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985292 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985326 6400 cpu_manager.go:414] "containerMap: removing container" podUID="68b35351-443b-40e5-b5a4-f548366bba8e" containerName="maas-api" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985333 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="68b35351-443b-40e5-b5a4-f548366bba8e" containerName="maas-api" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985343 6400 cpu_manager.go:414] "containerMap: removing container" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985350 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985362 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985369 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985407 6400 cpu_manager.go:414] "containerMap: removing container" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985414 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985441 6400 cpu_manager.go:414] "containerMap: removing container" podUID="97637696-5890-4712-b677-f9ef36fdf467" containerName="main" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985447 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97637696-5890-4712-b677-f9ef36fdf467" containerName="main" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985464 6400 cpu_manager.go:414] "containerMap: removing container" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" Oct 30 16:59:44.985470 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985470 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" Oct 30 16:59:44.986434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985651 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="628cf0ef-f9e0-4469-a4aa-5528721b5818" containerName="istio-proxy" Oct 30 16:59:44.986434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985665 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="68b35351-443b-40e5-b5a4-f548366bba8e" containerName="maas-api" Oct 30 16:59:44.986434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985675 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" Oct 30 16:59:44.986434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985690 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="745f06d2-beda-4d0f-8f95-322d0ea28329" containerName="istio-proxy" Oct 30 16:59:44.986434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985702 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="97637696-5890-4712-b677-f9ef36fdf467" containerName="main" Oct 30 16:59:44.986434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985715 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="35adff5a-315a-4e33-922c-ec81f3091a8f" containerName="istio-proxy" Oct 30 16:59:44.986434 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.985730 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="605b7c33-6b21-4aa8-bfa4-c85298b8b717" containerName="istio-proxy" Oct 30 16:59:44.991137 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:44.991112 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.007585 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.007559 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w0-9e533f-serving-cert\"" Oct 30 16:59:45.017205 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.017179 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb"] Oct 30 16:59:45.028969 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.028941 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4653719-1cc9-451a-8634-fa8924d6ac41-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.029218 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.029191 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.029267 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.029240 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg6bf\" (UniqueName: \"kubernetes.io/projected/a4653719-1cc9-451a-8634-fa8924d6ac41-kube-api-access-fg6bf\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.029366 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.029343 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.131038 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.130952 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.131425 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:45.131099 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-serving-cert: secret "maas-api-e2e-shared-b-w0-9e533f-serving-cert" not found Oct 30 16:59:45.131425 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.131116 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4653719-1cc9-451a-8634-fa8924d6ac41-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.131425 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:45.131158 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-maas-api-tls podName:a4653719-1cc9-451a-8634-fa8924d6ac41 nodeName:}" failed. No retries permitted until 2026-10-30 16:59:45.631142423 +0000 UTC m=+2294.781423389 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-maas-api-tls") pod "maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" (UID: "a4653719-1cc9-451a-8634-fa8924d6ac41") : secret "maas-api-e2e-shared-b-w0-9e533f-serving-cert" not found Oct 30 16:59:45.131425 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.131290 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.131425 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.131318 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg6bf\" (UniqueName: \"kubernetes.io/projected/a4653719-1cc9-451a-8634-fa8924d6ac41-kube-api-access-fg6bf\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.131802 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.131785 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4653719-1cc9-451a-8634-fa8924d6ac41-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.133911 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.133885 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-metrics-certs\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.156018 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.155994 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg6bf\" (UniqueName: \"kubernetes.io/projected/a4653719-1cc9-451a-8634-fa8924d6ac41-kube-api-access-fg6bf\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.637188 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.637157 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.639861 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.639836 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-maas-api-tls\") pod \"maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:45.909691 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:45.909585 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:46.243239 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:46.243206 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb"] Oct 30 16:59:46.243727 ip-10-0-128-181 kubenswrapper[6400]: W1030 16:59:46.243703 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4653719_1cc9_451a_8634_fa8924d6ac41.slice/crio-fba88f9ffe60caed3161dabff60adfc69bd10eb42efd86a2f9359d42ce231307 WatchSource:0}: Error finding container fba88f9ffe60caed3161dabff60adfc69bd10eb42efd86a2f9359d42ce231307: Status 404 returned error can't find the container with id fba88f9ffe60caed3161dabff60adfc69bd10eb42efd86a2f9359d42ce231307 Oct 30 16:59:46.773372 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:46.773333 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s"] Oct 30 16:59:46.773593 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:46.773562 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" podUID="fb133fc0-29a7-4a23-b69c-66e10cacf293" containerName="maas-api" containerID="cri-o://78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff" gracePeriod=30 Oct 30 16:59:46.809176 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:46.809064 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" event={"ID":"a4653719-1cc9-451a-8634-fa8924d6ac41","Type":"ContainerStarted","Data":"faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680"} Oct 30 16:59:46.809176 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:46.809142 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:46.809359 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:46.809337 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" event={"ID":"a4653719-1cc9-451a-8634-fa8924d6ac41","Type":"ContainerStarted","Data":"fba88f9ffe60caed3161dabff60adfc69bd10eb42efd86a2f9359d42ce231307"} Oct 30 16:59:46.876141 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:46.876038 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" podStartSLOduration=2.505014095 podStartE2EDuration="2.876023204s" totalImagesPullingTime="371.009109ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 16:59:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 16:59:46.873916335 +0000 UTC m=+2296.024197323" watchObservedRunningTime="2026-10-30 16:59:46.876023204 +0000 UTC m=+2296.026304191" Oct 30 16:59:47.209628 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.207544 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:59:47.290096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.288535 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb133fc0-29a7-4a23-b69c-66e10cacf293-kube-api-access-cs7kp\" (UniqueName: \"kubernetes.io/projected/fb133fc0-29a7-4a23-b69c-66e10cacf293-kube-api-access-cs7kp\") pod \"fb133fc0-29a7-4a23-b69c-66e10cacf293\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " Oct 30 16:59:47.290096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.288640 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb133fc0-29a7-4a23-b69c-66e10cacf293-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb133fc0-29a7-4a23-b69c-66e10cacf293-openshift-service-ca\") pod \"fb133fc0-29a7-4a23-b69c-66e10cacf293\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " Oct 30 16:59:47.290096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.288731 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-metrics-certs\") pod \"fb133fc0-29a7-4a23-b69c-66e10cacf293\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " Oct 30 16:59:47.290096 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.288851 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-maas-api-tls\") pod \"fb133fc0-29a7-4a23-b69c-66e10cacf293\" (UID: \"fb133fc0-29a7-4a23-b69c-66e10cacf293\") " Oct 30 16:59:47.296663 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.293959 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb133fc0-29a7-4a23-b69c-66e10cacf293-openshift-service-ca" pod "fb133fc0-29a7-4a23-b69c-66e10cacf293" (UID: "fb133fc0-29a7-4a23-b69c-66e10cacf293"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 16:59:47.296857 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.296830 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb133fc0-29a7-4a23-b69c-66e10cacf293-kube-api-access-cs7kp" pod "fb133fc0-29a7-4a23-b69c-66e10cacf293" (UID: "fb133fc0-29a7-4a23-b69c-66e10cacf293"). InnerVolumeSpecName "kube-api-access-cs7kp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 16:59:47.305974 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.305944 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-metrics-certs" pod "fb133fc0-29a7-4a23-b69c-66e10cacf293" (UID: "fb133fc0-29a7-4a23-b69c-66e10cacf293"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:47.306123 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.305953 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-maas-api-tls" pod "fb133fc0-29a7-4a23-b69c-66e10cacf293" (UID: "fb133fc0-29a7-4a23-b69c-66e10cacf293"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 16:59:47.391204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.391171 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:47.391204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.391202 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs7kp\" (UniqueName: \"kubernetes.io/projected/fb133fc0-29a7-4a23-b69c-66e10cacf293-kube-api-access-cs7kp\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:47.391204 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.391212 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb133fc0-29a7-4a23-b69c-66e10cacf293-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:47.391446 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.391221 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/fb133fc0-29a7-4a23-b69c-66e10cacf293-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 16:59:47.820940 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.820907 6400 generic.go:356] "Generic (PLEG): container finished" podID="fb133fc0-29a7-4a23-b69c-66e10cacf293" containerID="78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff" exitCode=0 Oct 30 16:59:47.821146 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.820958 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" Oct 30 16:59:47.821146 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.820983 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" event={"ID":"fb133fc0-29a7-4a23-b69c-66e10cacf293","Type":"ContainerDied","Data":"78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff"} Oct 30 16:59:47.821146 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.821036 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s" event={"ID":"fb133fc0-29a7-4a23-b69c-66e10cacf293","Type":"ContainerDied","Data":"0ae3825bab236016d22f3496c2a59036f1233317f64456666d3145248804950a"} Oct 30 16:59:47.821146 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.821059 6400 scope.go:122] "RemoveContainer" containerID="78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff" Oct 30 16:59:47.840079 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.840053 6400 scope.go:122] "RemoveContainer" containerID="78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff" Oct 30 16:59:47.840536 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:47.840499 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff\": container with ID starting with 78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff not found: ID does not exist" containerID="78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff" Oct 30 16:59:47.840665 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.840557 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff"} err="failed to get container status \"78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff\": rpc error: code = NotFound desc = could not find container \"78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff\": container with ID starting with 78f41baa651f7652a1f82cf3c230ccab6e2fb3460b6c4883ff7665900de638ff not found: ID does not exist" Oct 30 16:59:47.873473 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.873440 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s"] Oct 30 16:59:47.880110 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:47.880087 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-668047-57d9dbb6c6-92x8s"] Oct 30 16:59:48.968819 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:48.968785 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb133fc0-29a7-4a23-b69c-66e10cacf293" path="/var/lib/kubelet/pods/fb133fc0-29a7-4a23-b69c-66e10cacf293/volumes" Oct 30 16:59:52.842626 ip-10-0-128-181 kubenswrapper[6400]: I1030 16:59:52.837087 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 16:59:54.600122 ip-10-0-128-181 kubenswrapper[6400]: E1030 16:59:54.600087 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:00:00.229892 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.229853 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h"] Oct 30 17:00:00.231387 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.231359 6400 cpu_manager.go:414] "containerMap: removing container" podUID="fb133fc0-29a7-4a23-b69c-66e10cacf293" containerName="maas-api" Oct 30 17:00:00.231387 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.231386 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb133fc0-29a7-4a23-b69c-66e10cacf293" containerName="maas-api" Oct 30 17:00:00.231555 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.231526 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7623e9c-0154-4059-bf52-027533652d9e" containerName="istio-proxy" Oct 30 17:00:00.231555 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.231540 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb133fc0-29a7-4a23-b69c-66e10cacf293" containerName="maas-api" Oct 30 17:00:00.237073 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.237051 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" Oct 30 17:00:00.255201 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.255172 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:00:00.274694 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.273182 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h"] Oct 30 17:00:00.283333 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.283285 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz"] Oct 30 17:00:00.288164 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.288141 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2"] Oct 30 17:00:00.288351 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.288296 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" Oct 30 17:00:00.292590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.292568 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" Oct 30 17:00:00.307308 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.305399 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c"] Oct 30 17:00:00.354210 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.354174 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2"] Oct 30 17:00:00.355637 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.354498 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" Oct 30 17:00:00.364039 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.363917 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mwpts\" (UniqueName: \"kubernetes.io/projected/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-kube-api-access-mwpts\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h\" (UID: \"8b26fddb-3e85-48f0-af47-ffeb5b512dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" Oct 30 17:00:00.365569 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.365535 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf131cbf-5e5a-4e53-b630-5bb585d90398-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz\" (UID: \"bf131cbf-5e5a-4e53-b630-5bb585d90398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" Oct 30 17:00:00.365916 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.365796 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccgpc\" (UniqueName: \"kubernetes.io/projected/fd855be1-393e-4132-b51a-e8b9e0aed09a-kube-api-access-ccgpc\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2\" (UID: \"fd855be1-393e-4132-b51a-e8b9e0aed09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" Oct 30 17:00:00.366141 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.366120 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dcrl\" (UniqueName: \"kubernetes.io/projected/bf131cbf-5e5a-4e53-b630-5bb585d90398-kube-api-access-6dcrl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz\" (UID: \"bf131cbf-5e5a-4e53-b630-5bb585d90398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" Oct 30 17:00:00.366292 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.366277 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h\" (UID: \"8b26fddb-3e85-48f0-af47-ffeb5b512dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" Oct 30 17:00:00.366466 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.366447 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd855be1-393e-4132-b51a-e8b9e0aed09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2\" (UID: \"fd855be1-393e-4132-b51a-e8b9e0aed09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" Oct 30 17:00:00.387862 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.387833 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz"] Oct 30 17:00:00.409558 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.409520 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k"] Oct 30 17:00:00.414997 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.414975 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" Oct 30 17:00:00.415699 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.415670 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c"] Oct 30 17:00:00.421374 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.421348 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82"] Oct 30 17:00:00.430092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.430071 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k"] Oct 30 17:00:00.430224 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.430178 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" Oct 30 17:00:00.432422 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.432399 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq"] Oct 30 17:00:00.441965 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.441937 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" Oct 30 17:00:00.443725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.443705 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82"] Oct 30 17:00:00.452648 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.450955 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq"] Oct 30 17:00:00.468370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.467988 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dcrl\" (UniqueName: \"kubernetes.io/projected/bf131cbf-5e5a-4e53-b630-5bb585d90398-kube-api-access-6dcrl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz\" (UID: \"bf131cbf-5e5a-4e53-b630-5bb585d90398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" Oct 30 17:00:00.468370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.468032 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02eb1598-1e89-4828-bc01-ce68c0b851fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82\" (UID: \"02eb1598-1e89-4828-bc01-ce68c0b851fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" Oct 30 17:00:00.468370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.468083 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h\" (UID: \"8b26fddb-3e85-48f0-af47-ffeb5b512dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" Oct 30 17:00:00.468370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.468147 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73890793-133c-4e7b-903e-5a185f8b0711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k\" (UID: \"73890793-133c-4e7b-903e-5a185f8b0711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" Oct 30 17:00:00.468370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.468178 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd855be1-393e-4132-b51a-e8b9e0aed09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2\" (UID: \"fd855be1-393e-4132-b51a-e8b9e0aed09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" Oct 30 17:00:00.468370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.468206 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxbk\" (UniqueName: \"kubernetes.io/projected/02eb1598-1e89-4828-bc01-ce68c0b851fb-kube-api-access-cfxbk\") pod \"maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82\" (UID: \"02eb1598-1e89-4828-bc01-ce68c0b851fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" Oct 30 17:00:00.468370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.468301 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105526c7-7445-4065-b65d-a51b7178d0ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c\" (UID: \"105526c7-7445-4065-b65d-a51b7178d0ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" Oct 30 17:00:00.469653 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.469030 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h\" (UID: \"8b26fddb-3e85-48f0-af47-ffeb5b512dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" Oct 30 17:00:00.469653 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.469616 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd855be1-393e-4132-b51a-e8b9e0aed09a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2\" (UID: \"fd855be1-393e-4132-b51a-e8b9e0aed09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" Oct 30 17:00:00.470386 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.470027 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89x8s\" (UniqueName: \"kubernetes.io/projected/73890793-133c-4e7b-903e-5a185f8b0711-kube-api-access-89x8s\") pod \"maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k\" (UID: \"73890793-133c-4e7b-903e-5a185f8b0711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" Oct 30 17:00:00.470386 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.470074 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mwpts\" (UniqueName: \"kubernetes.io/projected/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-kube-api-access-mwpts\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h\" (UID: \"8b26fddb-3e85-48f0-af47-ffeb5b512dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" Oct 30 17:00:00.470386 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.470180 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf131cbf-5e5a-4e53-b630-5bb585d90398-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz\" (UID: \"bf131cbf-5e5a-4e53-b630-5bb585d90398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" Oct 30 17:00:00.470386 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.470226 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27k8m\" (UniqueName: \"kubernetes.io/projected/105526c7-7445-4065-b65d-a51b7178d0ed-kube-api-access-27k8m\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c\" (UID: \"105526c7-7445-4065-b65d-a51b7178d0ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" Oct 30 17:00:00.470386 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.470277 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccgpc\" (UniqueName: \"kubernetes.io/projected/fd855be1-393e-4132-b51a-e8b9e0aed09a-kube-api-access-ccgpc\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2\" (UID: \"fd855be1-393e-4132-b51a-e8b9e0aed09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" Oct 30 17:00:00.471914 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.471142 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf131cbf-5e5a-4e53-b630-5bb585d90398-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz\" (UID: \"bf131cbf-5e5a-4e53-b630-5bb585d90398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" Oct 30 17:00:00.493720 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.493693 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccgpc\" (UniqueName: \"kubernetes.io/projected/fd855be1-393e-4132-b51a-e8b9e0aed09a-kube-api-access-ccgpc\") pod \"maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2\" (UID: \"fd855be1-393e-4132-b51a-e8b9e0aed09a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" Oct 30 17:00:00.493899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.493829 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mwpts\" (UniqueName: \"kubernetes.io/projected/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-kube-api-access-mwpts\") pod \"maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h\" (UID: \"8b26fddb-3e85-48f0-af47-ffeb5b512dc2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" Oct 30 17:00:00.498978 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.498947 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dcrl\" (UniqueName: \"kubernetes.io/projected/bf131cbf-5e5a-4e53-b630-5bb585d90398-kube-api-access-6dcrl\") pod \"maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz\" (UID: \"bf131cbf-5e5a-4e53-b630-5bb585d90398\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" Oct 30 17:00:00.554361 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.554331 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" Oct 30 17:00:00.571686 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.571642 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73890793-133c-4e7b-903e-5a185f8b0711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k\" (UID: \"73890793-133c-4e7b-903e-5a185f8b0711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" Oct 30 17:00:00.571899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.571698 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxbk\" (UniqueName: \"kubernetes.io/projected/02eb1598-1e89-4828-bc01-ce68c0b851fb-kube-api-access-cfxbk\") pod \"maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82\" (UID: \"02eb1598-1e89-4828-bc01-ce68c0b851fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" Oct 30 17:00:00.571899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.571823 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db07007a-80e3-40c7-9385-070440683859-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq\" (UID: \"db07007a-80e3-40c7-9385-070440683859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" Oct 30 17:00:00.571899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.571857 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105526c7-7445-4065-b65d-a51b7178d0ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c\" (UID: \"105526c7-7445-4065-b65d-a51b7178d0ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" Oct 30 17:00:00.572069 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.571983 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89x8s\" (UniqueName: \"kubernetes.io/projected/73890793-133c-4e7b-903e-5a185f8b0711-kube-api-access-89x8s\") pod \"maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k\" (UID: \"73890793-133c-4e7b-903e-5a185f8b0711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" Oct 30 17:00:00.572121 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.572111 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27k8m\" (UniqueName: \"kubernetes.io/projected/105526c7-7445-4065-b65d-a51b7178d0ed-kube-api-access-27k8m\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c\" (UID: \"105526c7-7445-4065-b65d-a51b7178d0ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" Oct 30 17:00:00.572221 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.572201 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02eb1598-1e89-4828-bc01-ce68c0b851fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82\" (UID: \"02eb1598-1e89-4828-bc01-ce68c0b851fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" Oct 30 17:00:00.572313 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.572293 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6kt5\" (UniqueName: \"kubernetes.io/projected/db07007a-80e3-40c7-9385-070440683859-kube-api-access-d6kt5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq\" (UID: \"db07007a-80e3-40c7-9385-070440683859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" Oct 30 17:00:00.572492 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.572452 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73890793-133c-4e7b-903e-5a185f8b0711-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k\" (UID: \"73890793-133c-4e7b-903e-5a185f8b0711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" Oct 30 17:00:00.573228 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.573202 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105526c7-7445-4065-b65d-a51b7178d0ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c\" (UID: \"105526c7-7445-4065-b65d-a51b7178d0ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" Oct 30 17:00:00.573937 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.573881 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02eb1598-1e89-4828-bc01-ce68c0b851fb-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82\" (UID: \"02eb1598-1e89-4828-bc01-ce68c0b851fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" Oct 30 17:00:00.605187 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.605109 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxbk\" (UniqueName: \"kubernetes.io/projected/02eb1598-1e89-4828-bc01-ce68c0b851fb-kube-api-access-cfxbk\") pod \"maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82\" (UID: \"02eb1598-1e89-4828-bc01-ce68c0b851fb\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" Oct 30 17:00:00.605812 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.605781 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27k8m\" (UniqueName: \"kubernetes.io/projected/105526c7-7445-4065-b65d-a51b7178d0ed-kube-api-access-27k8m\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c\" (UID: \"105526c7-7445-4065-b65d-a51b7178d0ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" Oct 30 17:00:00.606400 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.606377 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89x8s\" (UniqueName: \"kubernetes.io/projected/73890793-133c-4e7b-903e-5a185f8b0711-kube-api-access-89x8s\") pod \"maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k\" (UID: \"73890793-133c-4e7b-903e-5a185f8b0711\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" Oct 30 17:00:00.674963 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.674930 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6kt5\" (UniqueName: \"kubernetes.io/projected/db07007a-80e3-40c7-9385-070440683859-kube-api-access-d6kt5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq\" (UID: \"db07007a-80e3-40c7-9385-070440683859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" Oct 30 17:00:00.675173 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.675013 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db07007a-80e3-40c7-9385-070440683859-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq\" (UID: \"db07007a-80e3-40c7-9385-070440683859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" Oct 30 17:00:00.675845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.675818 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db07007a-80e3-40c7-9385-070440683859-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq\" (UID: \"db07007a-80e3-40c7-9385-070440683859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" Oct 30 17:00:00.686949 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.686916 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" Oct 30 17:00:00.692097 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.692070 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" Oct 30 17:00:00.698746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.698723 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6kt5\" (UniqueName: \"kubernetes.io/projected/db07007a-80e3-40c7-9385-070440683859-kube-api-access-d6kt5\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq\" (UID: \"db07007a-80e3-40c7-9385-070440683859\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" Oct 30 17:00:00.711503 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.711473 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" Oct 30 17:00:00.732680 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.732649 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" Oct 30 17:00:00.766183 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.766153 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" Oct 30 17:00:00.775500 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.775298 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" Oct 30 17:00:00.882663 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:00:00.882620 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b26fddb_3e85_48f0_af47_ffeb5b512dc2.slice/crio-1a61ec6c7b707f6a2aedebe2d84ed591fdd87ff053c0855901c25cdd78a1fc22 WatchSource:0}: Error finding container 1a61ec6c7b707f6a2aedebe2d84ed591fdd87ff053c0855901c25cdd78a1fc22: Status 404 returned error can't find the container with id 1a61ec6c7b707f6a2aedebe2d84ed591fdd87ff053c0855901c25cdd78a1fc22 Oct 30 17:00:00.883680 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.883318 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h"] Oct 30 17:00:00.959388 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:00.959346 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" event={"ID":"8b26fddb-3e85-48f0-af47-ffeb5b512dc2","Type":"ContainerStarted","Data":"1a61ec6c7b707f6a2aedebe2d84ed591fdd87ff053c0855901c25cdd78a1fc22"} Oct 30 17:00:01.022992 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.022348 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz"] Oct 30 17:00:01.053032 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.052996 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c"] Oct 30 17:00:01.111134 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.110999 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k"] Oct 30 17:00:01.115374 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.115346 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2"] Oct 30 17:00:01.119192 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:00:01.119163 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd855be1_393e_4132_b51a_e8b9e0aed09a.slice/crio-db1a478986cce3b2944a448ab40ca0ebd71264cefd8c0ddca528c944e33f174d WatchSource:0}: Error finding container db1a478986cce3b2944a448ab40ca0ebd71264cefd8c0ddca528c944e33f174d: Status 404 returned error can't find the container with id db1a478986cce3b2944a448ab40ca0ebd71264cefd8c0ddca528c944e33f174d Oct 30 17:00:01.126687 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.126664 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82"] Oct 30 17:00:01.130434 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:00:01.130409 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02eb1598_1e89_4828_bc01_ce68c0b851fb.slice/crio-faa93385a80fc1a662fb5857c3804bd5627c28a2863ea7cff8d5ef5ccb33f97c WatchSource:0}: Error finding container faa93385a80fc1a662fb5857c3804bd5627c28a2863ea7cff8d5ef5ccb33f97c: Status 404 returned error can't find the container with id faa93385a80fc1a662fb5857c3804bd5627c28a2863ea7cff8d5ef5ccb33f97c Oct 30 17:00:01.204955 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.204923 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq"] Oct 30 17:00:01.205901 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:00:01.205873 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb07007a_80e3_40c7_9385_070440683859.slice/crio-974a1027f28e6ac6d76c7a38e7017624812a7770143fa212d33126252bea0b5b WatchSource:0}: Error finding container 974a1027f28e6ac6d76c7a38e7017624812a7770143fa212d33126252bea0b5b: Status 404 returned error can't find the container with id 974a1027f28e6ac6d76c7a38e7017624812a7770143fa212d33126252bea0b5b Oct 30 17:00:01.970142 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.970048 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" event={"ID":"02eb1598-1e89-4828-bc01-ce68c0b851fb","Type":"ContainerStarted","Data":"e2ddb4630f4bd5e9ad899abc423cff2be33027cf06ae0d06a7bf7f4fe9ef885d"} Oct 30 17:00:01.970142 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.970136 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" event={"ID":"02eb1598-1e89-4828-bc01-ce68c0b851fb","Type":"ContainerStarted","Data":"faa93385a80fc1a662fb5857c3804bd5627c28a2863ea7cff8d5ef5ccb33f97c"} Oct 30 17:00:01.971945 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.971916 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" event={"ID":"8b26fddb-3e85-48f0-af47-ffeb5b512dc2","Type":"ContainerStarted","Data":"3ba6489554704c279c8260c181d520dc338c78fe5a95f934f5dba8280cfdee2b"} Oct 30 17:00:01.973553 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.973526 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" event={"ID":"73890793-133c-4e7b-903e-5a185f8b0711","Type":"ContainerStarted","Data":"2120174a4deac632936ade306ab0b06548806bff98623782553818b973a936fb"} Oct 30 17:00:01.973691 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.973573 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" event={"ID":"73890793-133c-4e7b-903e-5a185f8b0711","Type":"ContainerStarted","Data":"607a010d1f3988e42cd91262c4d492e10014662e9a11121268c1c1ed5b12d92b"} Oct 30 17:00:01.975972 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.975943 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" event={"ID":"db07007a-80e3-40c7-9385-070440683859","Type":"ContainerStarted","Data":"5fb839c54ab45a638e4c6932ad75fa4472ad0baa95aaa6565befbb57158ba88f"} Oct 30 17:00:01.976085 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.976000 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" event={"ID":"db07007a-80e3-40c7-9385-070440683859","Type":"ContainerStarted","Data":"974a1027f28e6ac6d76c7a38e7017624812a7770143fa212d33126252bea0b5b"} Oct 30 17:00:01.977758 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.977735 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" event={"ID":"bf131cbf-5e5a-4e53-b630-5bb585d90398","Type":"ContainerStarted","Data":"e578769201243095de601d73bd93b39b9500b6524d486701156c0f9a25d56069"} Oct 30 17:00:01.977840 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.977777 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" event={"ID":"bf131cbf-5e5a-4e53-b630-5bb585d90398","Type":"ContainerStarted","Data":"acffe5408e5a897961ce11e765308f7d634a8a73f9bf43da9ba3e19da5c07bed"} Oct 30 17:00:01.979278 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.979258 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" event={"ID":"105526c7-7445-4065-b65d-a51b7178d0ed","Type":"ContainerStarted","Data":"c5de69c6df83ac96ab6759efd20310fe8673016c6a87ecfbd9c62f450ed9f58c"} Oct 30 17:00:01.979364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.979301 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" event={"ID":"105526c7-7445-4065-b65d-a51b7178d0ed","Type":"ContainerStarted","Data":"6591acd2491b9837b766b78661add286081086ba85455b8e8981cf9ed040bc20"} Oct 30 17:00:01.980712 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.980691 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" event={"ID":"fd855be1-393e-4132-b51a-e8b9e0aed09a","Type":"ContainerStarted","Data":"5a4c0aa4743fe5ed2f36a27faf56e4fef28f66eb00704dee9f7545f54cca4d9b"} Oct 30 17:00:01.980817 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:01.980726 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" event={"ID":"fd855be1-393e-4132-b51a-e8b9e0aed09a","Type":"ContainerStarted","Data":"db1a478986cce3b2944a448ab40ca0ebd71264cefd8c0ddca528c944e33f174d"} Oct 30 17:00:02.015336 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.015284 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" podStartSLOduration=2.015270475 podStartE2EDuration="2.015270475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.00911485 +0000 UTC m=+2311.159395838" watchObservedRunningTime="2026-10-30 17:00:02.015270475 +0000 UTC m=+2311.165551459" Oct 30 17:00:02.049900 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.049848 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" podStartSLOduration=2.049834806 podStartE2EDuration="2.049834806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.049356854 +0000 UTC m=+2311.199637844" watchObservedRunningTime="2026-10-30 17:00:02.049834806 +0000 UTC m=+2311.200115794" Oct 30 17:00:02.092247 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.092195 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" podStartSLOduration=2.092181049 podStartE2EDuration="2.092181049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.091141647 +0000 UTC m=+2311.241422636" watchObservedRunningTime="2026-10-30 17:00:02.092181049 +0000 UTC m=+2311.242462076" Oct 30 17:00:02.180027 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.179969 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" podStartSLOduration=2.179953831 podStartE2EDuration="2.179953831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.133942953 +0000 UTC m=+2311.284223940" watchObservedRunningTime="2026-10-30 17:00:02.179953831 +0000 UTC m=+2311.330234819" Oct 30 17:00:02.181191 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.181153 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" podStartSLOduration=2.181145494 podStartE2EDuration="2.181145494s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.17882341 +0000 UTC m=+2311.329104398" watchObservedRunningTime="2026-10-30 17:00:02.181145494 +0000 UTC m=+2311.331426481" Oct 30 17:00:02.223949 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.223869 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" podStartSLOduration=2.223855282 podStartE2EDuration="2.223855282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.220465509 +0000 UTC m=+2311.370746499" watchObservedRunningTime="2026-10-30 17:00:02.223855282 +0000 UTC m=+2311.374136269" Oct 30 17:00:02.265665 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.265570 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" podStartSLOduration=2.265552443 podStartE2EDuration="2.265552443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:00:02.263709213 +0000 UTC m=+2311.413990210" watchObservedRunningTime="2026-10-30 17:00:02.265552443 +0000 UTC m=+2311.415833428" Oct 30 17:00:02.989782 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.989748 6400 generic.go:356] "Generic (PLEG): container finished" podID="73890793-133c-4e7b-903e-5a185f8b0711" containerID="2120174a4deac632936ade306ab0b06548806bff98623782553818b973a936fb" exitCode=60 Oct 30 17:00:02.990256 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.989825 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" event={"ID":"73890793-133c-4e7b-903e-5a185f8b0711","Type":"ContainerDied","Data":"2120174a4deac632936ade306ab0b06548806bff98623782553818b973a936fb"} Oct 30 17:00:02.990256 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.990168 6400 scope.go:122] "RemoveContainer" containerID="2120174a4deac632936ade306ab0b06548806bff98623782553818b973a936fb" Oct 30 17:00:02.991525 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.991496 6400 generic.go:356] "Generic (PLEG): container finished" podID="db07007a-80e3-40c7-9385-070440683859" containerID="5fb839c54ab45a638e4c6932ad75fa4472ad0baa95aaa6565befbb57158ba88f" exitCode=60 Oct 30 17:00:02.991662 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.991579 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" event={"ID":"db07007a-80e3-40c7-9385-070440683859","Type":"ContainerDied","Data":"5fb839c54ab45a638e4c6932ad75fa4472ad0baa95aaa6565befbb57158ba88f"} Oct 30 17:00:02.991895 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.991878 6400 scope.go:122] "RemoveContainer" containerID="5fb839c54ab45a638e4c6932ad75fa4472ad0baa95aaa6565befbb57158ba88f" Oct 30 17:00:02.993500 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.993481 6400 generic.go:356] "Generic (PLEG): container finished" podID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerID="e578769201243095de601d73bd93b39b9500b6524d486701156c0f9a25d56069" exitCode=60 Oct 30 17:00:02.993592 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.993572 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" event={"ID":"bf131cbf-5e5a-4e53-b630-5bb585d90398","Type":"ContainerDied","Data":"e578769201243095de601d73bd93b39b9500b6524d486701156c0f9a25d56069"} Oct 30 17:00:02.993791 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.993773 6400 scope.go:122] "RemoveContainer" containerID="e578769201243095de601d73bd93b39b9500b6524d486701156c0f9a25d56069" Oct 30 17:00:02.995521 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.995423 6400 generic.go:356] "Generic (PLEG): container finished" podID="105526c7-7445-4065-b65d-a51b7178d0ed" containerID="c5de69c6df83ac96ab6759efd20310fe8673016c6a87ecfbd9c62f450ed9f58c" exitCode=60 Oct 30 17:00:02.995521 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.995470 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" event={"ID":"105526c7-7445-4065-b65d-a51b7178d0ed","Type":"ContainerDied","Data":"c5de69c6df83ac96ab6759efd20310fe8673016c6a87ecfbd9c62f450ed9f58c"} Oct 30 17:00:02.995689 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.995673 6400 scope.go:122] "RemoveContainer" containerID="c5de69c6df83ac96ab6759efd20310fe8673016c6a87ecfbd9c62f450ed9f58c" Oct 30 17:00:02.997082 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.997063 6400 generic.go:356] "Generic (PLEG): container finished" podID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerID="5a4c0aa4743fe5ed2f36a27faf56e4fef28f66eb00704dee9f7545f54cca4d9b" exitCode=60 Oct 30 17:00:02.997156 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.997139 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" event={"ID":"fd855be1-393e-4132-b51a-e8b9e0aed09a","Type":"ContainerDied","Data":"5a4c0aa4743fe5ed2f36a27faf56e4fef28f66eb00704dee9f7545f54cca4d9b"} Oct 30 17:00:02.997316 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.997302 6400 scope.go:122] "RemoveContainer" containerID="5a4c0aa4743fe5ed2f36a27faf56e4fef28f66eb00704dee9f7545f54cca4d9b" Oct 30 17:00:02.999018 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.998995 6400 generic.go:356] "Generic (PLEG): container finished" podID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerID="e2ddb4630f4bd5e9ad899abc423cff2be33027cf06ae0d06a7bf7f4fe9ef885d" exitCode=60 Oct 30 17:00:02.999307 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.999290 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" event={"ID":"02eb1598-1e89-4828-bc01-ce68c0b851fb","Type":"ContainerDied","Data":"e2ddb4630f4bd5e9ad899abc423cff2be33027cf06ae0d06a7bf7f4fe9ef885d"} Oct 30 17:00:02.999425 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:02.999411 6400 scope.go:122] "RemoveContainer" containerID="e2ddb4630f4bd5e9ad899abc423cff2be33027cf06ae0d06a7bf7f4fe9ef885d" Oct 30 17:00:03.001394 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:03.001372 6400 generic.go:356] "Generic (PLEG): container finished" podID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerID="3ba6489554704c279c8260c181d520dc338c78fe5a95f934f5dba8280cfdee2b" exitCode=60 Oct 30 17:00:03.001487 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:03.001442 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" event={"ID":"8b26fddb-3e85-48f0-af47-ffeb5b512dc2","Type":"ContainerDied","Data":"3ba6489554704c279c8260c181d520dc338c78fe5a95f934f5dba8280cfdee2b"} Oct 30 17:00:03.001739 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:03.001721 6400 scope.go:122] "RemoveContainer" containerID="3ba6489554704c279c8260c181d520dc338c78fe5a95f934f5dba8280cfdee2b" Oct 30 17:00:04.016677 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:04.016643 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" event={"ID":"02eb1598-1e89-4828-bc01-ce68c0b851fb","Type":"ContainerStarted","Data":"c4eafc770564f550795a1818a462538cdff19b4cc251bcabd08d0802dd5aca32"} Oct 30 17:00:04.019746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:04.019682 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" event={"ID":"8b26fddb-3e85-48f0-af47-ffeb5b512dc2","Type":"ContainerStarted","Data":"19697e88fae97ccc447220cccb3e19db0365e27f2197a31cf443313261ee1747"} Oct 30 17:00:04.022143 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:04.022099 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" event={"ID":"73890793-133c-4e7b-903e-5a185f8b0711","Type":"ContainerStarted","Data":"bc558b18a9d0ca67fd9016ab1e172174167ab0c0f864dd9bdf5d44be05949e8f"} Oct 30 17:00:04.024703 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:04.024661 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" event={"ID":"db07007a-80e3-40c7-9385-070440683859","Type":"ContainerStarted","Data":"23f625e82a8872587311d1fa85581a5beb3b16f5e67c7a70454be4d3aec6b7e0"} Oct 30 17:00:04.028208 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:04.028181 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" event={"ID":"bf131cbf-5e5a-4e53-b630-5bb585d90398","Type":"ContainerStarted","Data":"ed8671324726edd30134a5f7f7f3885028f9ebf18122672b2260dd8be7a5159c"} Oct 30 17:00:04.033297 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:04.033269 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" event={"ID":"105526c7-7445-4065-b65d-a51b7178d0ed","Type":"ContainerStarted","Data":"06692ae53af065d4bfe423d8e3dd1d010af49d0a661dee1dc4f5a2304939d563"} Oct 30 17:00:04.038428 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:04.038396 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" event={"ID":"fd855be1-393e-4132-b51a-e8b9e0aed09a","Type":"ContainerStarted","Data":"986db1d9d83b31ddff736a5dcc7bbaf6a1235323f58b9d37212dc9811a032391"} Oct 30 17:00:05.050398 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.050359 6400 generic.go:356] "Generic (PLEG): container finished" podID="db07007a-80e3-40c7-9385-070440683859" containerID="23f625e82a8872587311d1fa85581a5beb3b16f5e67c7a70454be4d3aec6b7e0" exitCode=60 Oct 30 17:00:05.050871 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.050442 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" event={"ID":"db07007a-80e3-40c7-9385-070440683859","Type":"ContainerDied","Data":"23f625e82a8872587311d1fa85581a5beb3b16f5e67c7a70454be4d3aec6b7e0"} Oct 30 17:00:05.050871 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.050499 6400 scope.go:122] "RemoveContainer" containerID="5fb839c54ab45a638e4c6932ad75fa4472ad0baa95aaa6565befbb57158ba88f" Oct 30 17:00:05.050871 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.050778 6400 scope.go:122] "RemoveContainer" containerID="23f625e82a8872587311d1fa85581a5beb3b16f5e67c7a70454be4d3aec6b7e0" Oct 30 17:00:05.051114 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:05.051088 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq_odh-ai-gateway-infra(db07007a-80e3-40c7-9385-070440683859)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" podUID="db07007a-80e3-40c7-9385-070440683859" Oct 30 17:00:05.052587 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.052565 6400 generic.go:356] "Generic (PLEG): container finished" podID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerID="ed8671324726edd30134a5f7f7f3885028f9ebf18122672b2260dd8be7a5159c" exitCode=60 Oct 30 17:00:05.052729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.052656 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" event={"ID":"bf131cbf-5e5a-4e53-b630-5bb585d90398","Type":"ContainerDied","Data":"ed8671324726edd30134a5f7f7f3885028f9ebf18122672b2260dd8be7a5159c"} Oct 30 17:00:05.052884 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.052862 6400 scope.go:122] "RemoveContainer" containerID="ed8671324726edd30134a5f7f7f3885028f9ebf18122672b2260dd8be7a5159c" Oct 30 17:00:05.053141 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:05.053118 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz_odh-ai-gateway-infra(bf131cbf-5e5a-4e53-b630-5bb585d90398)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" Oct 30 17:00:05.054869 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.054848 6400 generic.go:356] "Generic (PLEG): container finished" podID="105526c7-7445-4065-b65d-a51b7178d0ed" containerID="06692ae53af065d4bfe423d8e3dd1d010af49d0a661dee1dc4f5a2304939d563" exitCode=60 Oct 30 17:00:05.055032 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.054899 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" event={"ID":"105526c7-7445-4065-b65d-a51b7178d0ed","Type":"ContainerDied","Data":"06692ae53af065d4bfe423d8e3dd1d010af49d0a661dee1dc4f5a2304939d563"} Oct 30 17:00:05.055236 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.055213 6400 scope.go:122] "RemoveContainer" containerID="06692ae53af065d4bfe423d8e3dd1d010af49d0a661dee1dc4f5a2304939d563" Oct 30 17:00:05.055523 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:05.055498 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c_odh-ai-gateway-infra(105526c7-7445-4065-b65d-a51b7178d0ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" Oct 30 17:00:05.057144 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.057121 6400 generic.go:356] "Generic (PLEG): container finished" podID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerID="986db1d9d83b31ddff736a5dcc7bbaf6a1235323f58b9d37212dc9811a032391" exitCode=60 Oct 30 17:00:05.057235 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.057203 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" event={"ID":"fd855be1-393e-4132-b51a-e8b9e0aed09a","Type":"ContainerDied","Data":"986db1d9d83b31ddff736a5dcc7bbaf6a1235323f58b9d37212dc9811a032391"} Oct 30 17:00:05.057418 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.057405 6400 scope.go:122] "RemoveContainer" containerID="986db1d9d83b31ddff736a5dcc7bbaf6a1235323f58b9d37212dc9811a032391" Oct 30 17:00:05.057681 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:05.057662 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2_odh-ai-gateway-infra(fd855be1-393e-4132-b51a-e8b9e0aed09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" Oct 30 17:00:05.059335 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.059314 6400 generic.go:356] "Generic (PLEG): container finished" podID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerID="c4eafc770564f550795a1818a462538cdff19b4cc251bcabd08d0802dd5aca32" exitCode=60 Oct 30 17:00:05.059421 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.059345 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" event={"ID":"02eb1598-1e89-4828-bc01-ce68c0b851fb","Type":"ContainerDied","Data":"c4eafc770564f550795a1818a462538cdff19b4cc251bcabd08d0802dd5aca32"} Oct 30 17:00:05.059635 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.059595 6400 scope.go:122] "RemoveContainer" containerID="c4eafc770564f550795a1818a462538cdff19b4cc251bcabd08d0802dd5aca32" Oct 30 17:00:05.059864 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:05.059847 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82_odh-ai-gateway-infra(02eb1598-1e89-4828-bc01-ce68c0b851fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" Oct 30 17:00:05.061434 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.061416 6400 generic.go:356] "Generic (PLEG): container finished" podID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerID="19697e88fae97ccc447220cccb3e19db0365e27f2197a31cf443313261ee1747" exitCode=60 Oct 30 17:00:05.061497 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.061449 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" event={"ID":"8b26fddb-3e85-48f0-af47-ffeb5b512dc2","Type":"ContainerDied","Data":"19697e88fae97ccc447220cccb3e19db0365e27f2197a31cf443313261ee1747"} Oct 30 17:00:05.061788 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.061762 6400 scope.go:122] "RemoveContainer" containerID="19697e88fae97ccc447220cccb3e19db0365e27f2197a31cf443313261ee1747" Oct 30 17:00:05.062044 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:05.062019 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h_odh-ai-gateway-infra(8b26fddb-3e85-48f0-af47-ffeb5b512dc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" Oct 30 17:00:05.068786 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.068677 6400 generic.go:356] "Generic (PLEG): container finished" podID="73890793-133c-4e7b-903e-5a185f8b0711" containerID="bc558b18a9d0ca67fd9016ab1e172174167ab0c0f864dd9bdf5d44be05949e8f" exitCode=60 Oct 30 17:00:05.068786 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.068729 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" event={"ID":"73890793-133c-4e7b-903e-5a185f8b0711","Type":"ContainerDied","Data":"bc558b18a9d0ca67fd9016ab1e172174167ab0c0f864dd9bdf5d44be05949e8f"} Oct 30 17:00:05.069098 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.069080 6400 scope.go:122] "RemoveContainer" containerID="bc558b18a9d0ca67fd9016ab1e172174167ab0c0f864dd9bdf5d44be05949e8f" Oct 30 17:00:05.069187 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.069169 6400 scope.go:122] "RemoveContainer" containerID="e578769201243095de601d73bd93b39b9500b6524d486701156c0f9a25d56069" Oct 30 17:00:05.069513 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:05.069477 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k_odh-ai-gateway-infra(73890793-133c-4e7b-903e-5a185f8b0711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" podUID="73890793-133c-4e7b-903e-5a185f8b0711" Oct 30 17:00:05.084562 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.084540 6400 scope.go:122] "RemoveContainer" containerID="c5de69c6df83ac96ab6759efd20310fe8673016c6a87ecfbd9c62f450ed9f58c" Oct 30 17:00:05.099020 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.098997 6400 scope.go:122] "RemoveContainer" containerID="5a4c0aa4743fe5ed2f36a27faf56e4fef28f66eb00704dee9f7545f54cca4d9b" Oct 30 17:00:05.112820 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.112800 6400 scope.go:122] "RemoveContainer" containerID="e2ddb4630f4bd5e9ad899abc423cff2be33027cf06ae0d06a7bf7f4fe9ef885d" Oct 30 17:00:05.126623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.126589 6400 scope.go:122] "RemoveContainer" containerID="3ba6489554704c279c8260c181d520dc338c78fe5a95f934f5dba8280cfdee2b" Oct 30 17:00:05.144831 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:05.144803 6400 scope.go:122] "RemoveContainer" containerID="2120174a4deac632936ade306ab0b06548806bff98623782553818b973a936fb" Oct 30 17:00:06.079371 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:06.079342 6400 scope.go:122] "RemoveContainer" containerID="c4eafc770564f550795a1818a462538cdff19b4cc251bcabd08d0802dd5aca32" Oct 30 17:00:06.079811 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:06.079673 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82_odh-ai-gateway-infra(02eb1598-1e89-4828-bc01-ce68c0b851fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" Oct 30 17:00:06.081092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:06.081057 6400 scope.go:122] "RemoveContainer" containerID="19697e88fae97ccc447220cccb3e19db0365e27f2197a31cf443313261ee1747" Oct 30 17:00:06.081390 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:06.081367 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h_odh-ai-gateway-infra(8b26fddb-3e85-48f0-af47-ffeb5b512dc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" Oct 30 17:00:06.082571 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:06.082546 6400 scope.go:122] "RemoveContainer" containerID="bc558b18a9d0ca67fd9016ab1e172174167ab0c0f864dd9bdf5d44be05949e8f" Oct 30 17:00:06.082809 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:06.082785 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k_odh-ai-gateway-infra(73890793-133c-4e7b-903e-5a185f8b0711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" podUID="73890793-133c-4e7b-903e-5a185f8b0711" Oct 30 17:00:06.084035 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:06.084019 6400 scope.go:122] "RemoveContainer" containerID="23f625e82a8872587311d1fa85581a5beb3b16f5e67c7a70454be4d3aec6b7e0" Oct 30 17:00:06.084242 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:06.084221 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq_odh-ai-gateway-infra(db07007a-80e3-40c7-9385-070440683859)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" podUID="db07007a-80e3-40c7-9385-070440683859" Oct 30 17:00:06.085531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:06.085515 6400 scope.go:122] "RemoveContainer" containerID="ed8671324726edd30134a5f7f7f3885028f9ebf18122672b2260dd8be7a5159c" Oct 30 17:00:06.085763 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:06.085744 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz_odh-ai-gateway-infra(bf131cbf-5e5a-4e53-b630-5bb585d90398)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" Oct 30 17:00:06.087070 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:06.087051 6400 scope.go:122] "RemoveContainer" containerID="06692ae53af065d4bfe423d8e3dd1d010af49d0a661dee1dc4f5a2304939d563" Oct 30 17:00:06.087310 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:06.087288 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c_odh-ai-gateway-infra(105526c7-7445-4065-b65d-a51b7178d0ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" Oct 30 17:00:06.088589 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:06.088574 6400 scope.go:122] "RemoveContainer" containerID="986db1d9d83b31ddff736a5dcc7bbaf6a1235323f58b9d37212dc9811a032391" Oct 30 17:00:06.088814 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:06.088797 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2_odh-ai-gateway-infra(fd855be1-393e-4132-b51a-e8b9e0aed09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" Oct 30 17:00:14.961370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:14.961341 6400 scope.go:122] "RemoveContainer" containerID="ed8671324726edd30134a5f7f7f3885028f9ebf18122672b2260dd8be7a5159c" Oct 30 17:00:15.190996 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.189990 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m"] Oct 30 17:00:15.195956 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.195933 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.217921 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.217852 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m"] Oct 30 17:00:15.223656 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.221966 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:00:15.300836 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.300797 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.301095 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.301071 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp9kq\" (UniqueName: \"kubernetes.io/projected/2dc73836-c587-4d59-8c93-3b53d3a12ba6-kube-api-access-zp9kq\") pod \"maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.301234 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.301211 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.402683 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.402649 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.402827 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.402729 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.402890 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.402862 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp9kq\" (UniqueName: \"kubernetes.io/projected/2dc73836-c587-4d59-8c93-3b53d3a12ba6-kube-api-access-zp9kq\") pod \"maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.403454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.403428 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.405238 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.405221 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.425919 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.425897 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp9kq\" (UniqueName: \"kubernetes.io/projected/2dc73836-c587-4d59-8c93-3b53d3a12ba6-kube-api-access-zp9kq\") pod \"maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.517453 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.517425 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:15.822787 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:00:15.822753 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc73836_c587_4d59_8c93_3b53d3a12ba6.slice/crio-b622dc64e314e97920bce2df5715e38ab28788f747602a8ebc9c663094fb3fb3 WatchSource:0}: Error finding container b622dc64e314e97920bce2df5715e38ab28788f747602a8ebc9c663094fb3fb3: Status 404 returned error can't find the container with id b622dc64e314e97920bce2df5715e38ab28788f747602a8ebc9c663094fb3fb3 Oct 30 17:00:15.824512 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:15.824482 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m"] Oct 30 17:00:16.174560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:16.174524 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" event={"ID":"2dc73836-c587-4d59-8c93-3b53d3a12ba6","Type":"ContainerStarted","Data":"b622dc64e314e97920bce2df5715e38ab28788f747602a8ebc9c663094fb3fb3"} Oct 30 17:00:16.176593 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:16.176569 6400 generic.go:356] "Generic (PLEG): container finished" podID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerID="00be22df726711ce06fe39bce2a96e8f31d4f1699d5b463fb732e63d48569e31" exitCode=60 Oct 30 17:00:16.176755 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:16.176654 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" event={"ID":"bf131cbf-5e5a-4e53-b630-5bb585d90398","Type":"ContainerDied","Data":"00be22df726711ce06fe39bce2a96e8f31d4f1699d5b463fb732e63d48569e31"} Oct 30 17:00:16.176755 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:16.176718 6400 scope.go:122] "RemoveContainer" containerID="ed8671324726edd30134a5f7f7f3885028f9ebf18122672b2260dd8be7a5159c" Oct 30 17:00:16.177147 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:16.177132 6400 scope.go:122] "RemoveContainer" containerID="00be22df726711ce06fe39bce2a96e8f31d4f1699d5b463fb732e63d48569e31" Oct 30 17:00:16.177509 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:16.177487 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz_odh-ai-gateway-infra(bf131cbf-5e5a-4e53-b630-5bb585d90398)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" Oct 30 17:00:16.960011 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:16.959980 6400 scope.go:122] "RemoveContainer" containerID="06692ae53af065d4bfe423d8e3dd1d010af49d0a661dee1dc4f5a2304939d563" Oct 30 17:00:16.960192 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:16.960174 6400 scope.go:122] "RemoveContainer" containerID="bc558b18a9d0ca67fd9016ab1e172174167ab0c0f864dd9bdf5d44be05949e8f" Oct 30 17:00:16.960344 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:16.960315 6400 scope.go:122] "RemoveContainer" containerID="986db1d9d83b31ddff736a5dcc7bbaf6a1235323f58b9d37212dc9811a032391" Oct 30 17:00:16.960406 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:16.960385 6400 scope.go:122] "RemoveContainer" containerID="19697e88fae97ccc447220cccb3e19db0365e27f2197a31cf443313261ee1747" Oct 30 17:00:17.185758 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.185717 6400 generic.go:356] "Generic (PLEG): container finished" podID="2dc73836-c587-4d59-8c93-3b53d3a12ba6" containerID="c0b350e60902475d660a2c52527b977f6d28290ff2b6d0c2931195115cece5c0" exitCode=0 Oct 30 17:00:17.185758 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.185754 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" event={"ID":"2dc73836-c587-4d59-8c93-3b53d3a12ba6","Type":"ContainerDied","Data":"c0b350e60902475d660a2c52527b977f6d28290ff2b6d0c2931195115cece5c0"} Oct 30 17:00:17.230912 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.230880 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz"] Oct 30 17:00:17.415575 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.415550 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" Oct 30 17:00:17.536596 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.536571 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf131cbf-5e5a-4e53-b630-5bb585d90398-kube-api-access-6dcrl\" (UniqueName: \"kubernetes.io/projected/bf131cbf-5e5a-4e53-b630-5bb585d90398-kube-api-access-6dcrl\") pod \"bf131cbf-5e5a-4e53-b630-5bb585d90398\" (UID: \"bf131cbf-5e5a-4e53-b630-5bb585d90398\") " Oct 30 17:00:17.536829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.536812 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf131cbf-5e5a-4e53-b630-5bb585d90398-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf131cbf-5e5a-4e53-b630-5bb585d90398-maas-api-service-ca\") pod \"bf131cbf-5e5a-4e53-b630-5bb585d90398\" (UID: \"bf131cbf-5e5a-4e53-b630-5bb585d90398\") " Oct 30 17:00:17.537268 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.537173 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf131cbf-5e5a-4e53-b630-5bb585d90398-maas-api-service-ca" pod "bf131cbf-5e5a-4e53-b630-5bb585d90398" (UID: "bf131cbf-5e5a-4e53-b630-5bb585d90398"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:17.538003 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.537363 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf131cbf-5e5a-4e53-b630-5bb585d90398-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:17.538812 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.538785 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf131cbf-5e5a-4e53-b630-5bb585d90398-kube-api-access-6dcrl" pod "bf131cbf-5e5a-4e53-b630-5bb585d90398" (UID: "bf131cbf-5e5a-4e53-b630-5bb585d90398"). InnerVolumeSpecName "kube-api-access-6dcrl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:17.638910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.638876 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dcrl\" (UniqueName: \"kubernetes.io/projected/bf131cbf-5e5a-4e53-b630-5bb585d90398-kube-api-access-6dcrl\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:17.961034 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.960963 6400 scope.go:122] "RemoveContainer" containerID="c4eafc770564f550795a1818a462538cdff19b4cc251bcabd08d0802dd5aca32" Oct 30 17:00:17.961207 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:17.961085 6400 scope.go:122] "RemoveContainer" containerID="23f625e82a8872587311d1fa85581a5beb3b16f5e67c7a70454be4d3aec6b7e0" Oct 30 17:00:18.197003 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.196969 6400 generic.go:356] "Generic (PLEG): container finished" podID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerID="8ae927b556c51c6e8b322b4e57a571e9947cfdec743525059ad9aec99d92dbc7" exitCode=60 Oct 30 17:00:18.197478 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.197040 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" event={"ID":"8b26fddb-3e85-48f0-af47-ffeb5b512dc2","Type":"ContainerDied","Data":"8ae927b556c51c6e8b322b4e57a571e9947cfdec743525059ad9aec99d92dbc7"} Oct 30 17:00:18.197478 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.197103 6400 scope.go:122] "RemoveContainer" containerID="19697e88fae97ccc447220cccb3e19db0365e27f2197a31cf443313261ee1747" Oct 30 17:00:18.197702 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.197593 6400 scope.go:122] "RemoveContainer" containerID="8ae927b556c51c6e8b322b4e57a571e9947cfdec743525059ad9aec99d92dbc7" Oct 30 17:00:18.198230 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:18.198097 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h_odh-ai-gateway-infra(8b26fddb-3e85-48f0-af47-ffeb5b512dc2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" Oct 30 17:00:18.199483 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.199462 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" event={"ID":"73890793-133c-4e7b-903e-5a185f8b0711","Type":"ContainerStarted","Data":"9b501532ed7bcd5bcc6e7f02d80fad2438738d1f47d5b128c0d03928b725917b"} Oct 30 17:00:18.200889 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.200862 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" Oct 30 17:00:18.200889 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.200872 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz" event={"ID":"bf131cbf-5e5a-4e53-b630-5bb585d90398","Type":"ContainerDied","Data":"acffe5408e5a897961ce11e765308f7d634a8a73f9bf43da9ba3e19da5c07bed"} Oct 30 17:00:18.203032 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.203013 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" event={"ID":"105526c7-7445-4065-b65d-a51b7178d0ed","Type":"ContainerStarted","Data":"dab85ad5ed591c0592db05fb80a215d0bf3b257d18476a8da881c446dcc78948"} Oct 30 17:00:18.205303 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.205283 6400 generic.go:356] "Generic (PLEG): container finished" podID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerID="eb807ccba0495fc1291c1bdd15e9498bae4f8623516b36806cfb11bd64e27fae" exitCode=60 Oct 30 17:00:18.205408 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.205358 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" event={"ID":"fd855be1-393e-4132-b51a-e8b9e0aed09a","Type":"ContainerDied","Data":"eb807ccba0495fc1291c1bdd15e9498bae4f8623516b36806cfb11bd64e27fae"} Oct 30 17:00:18.205730 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.205715 6400 scope.go:122] "RemoveContainer" containerID="eb807ccba0495fc1291c1bdd15e9498bae4f8623516b36806cfb11bd64e27fae" Oct 30 17:00:18.206044 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:18.206023 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2_odh-ai-gateway-infra(fd855be1-393e-4132-b51a-e8b9e0aed09a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" Oct 30 17:00:18.426345 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.426320 6400 scope.go:122] "RemoveContainer" containerID="00be22df726711ce06fe39bce2a96e8f31d4f1699d5b463fb732e63d48569e31" Oct 30 17:00:18.438530 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.438506 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:18.439227 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.439212 6400 scope.go:122] "RemoveContainer" containerID="986db1d9d83b31ddff736a5dcc7bbaf6a1235323f58b9d37212dc9811a032391" Oct 30 17:00:18.529483 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.529443 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz"] Oct 30 17:00:18.546275 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.546243 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w4-7c4b84-29889660-hzdjz"] Oct 30 17:00:18.551690 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.551644 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-openshift-ca\") pod \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " Oct 30 17:00:18.551842 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.551826 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-xks-ca\") pod \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " Oct 30 17:00:18.552022 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.552003 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc73836-c587-4d59-8c93-3b53d3a12ba6-kube-api-access-zp9kq\" (UniqueName: \"kubernetes.io/projected/2dc73836-c587-4d59-8c93-3b53d3a12ba6-kube-api-access-zp9kq\") pod \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\" (UID: \"2dc73836-c587-4d59-8c93-3b53d3a12ba6\") " Oct 30 17:00:18.552097 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.552063 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-openshift-ca" pod "2dc73836-c587-4d59-8c93-3b53d3a12ba6" (UID: "2dc73836-c587-4d59-8c93-3b53d3a12ba6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:18.552575 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.552555 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:18.554223 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.554202 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-xks-ca" pod "2dc73836-c587-4d59-8c93-3b53d3a12ba6" (UID: "2dc73836-c587-4d59-8c93-3b53d3a12ba6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:18.554299 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.554286 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc73836-c587-4d59-8c93-3b53d3a12ba6-kube-api-access-zp9kq" pod "2dc73836-c587-4d59-8c93-3b53d3a12ba6" (UID: "2dc73836-c587-4d59-8c93-3b53d3a12ba6"). InnerVolumeSpecName "kube-api-access-zp9kq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:18.653979 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.653952 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2dc73836-c587-4d59-8c93-3b53d3a12ba6-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:18.653979 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.653979 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp9kq\" (UniqueName: \"kubernetes.io/projected/2dc73836-c587-4d59-8c93-3b53d3a12ba6-kube-api-access-zp9kq\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:18.968563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:18.968469 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" path="/var/lib/kubelet/pods/bf131cbf-5e5a-4e53-b630-5bb585d90398/volumes" Oct 30 17:00:19.224351 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.224284 6400 generic.go:356] "Generic (PLEG): container finished" podID="73890793-133c-4e7b-903e-5a185f8b0711" containerID="9b501532ed7bcd5bcc6e7f02d80fad2438738d1f47d5b128c0d03928b725917b" exitCode=60 Oct 30 17:00:19.226117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.224364 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" event={"ID":"73890793-133c-4e7b-903e-5a185f8b0711","Type":"ContainerDied","Data":"9b501532ed7bcd5bcc6e7f02d80fad2438738d1f47d5b128c0d03928b725917b"} Oct 30 17:00:19.226117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.224448 6400 scope.go:122] "RemoveContainer" containerID="bc558b18a9d0ca67fd9016ab1e172174167ab0c0f864dd9bdf5d44be05949e8f" Oct 30 17:00:19.226117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.224979 6400 scope.go:122] "RemoveContainer" containerID="9b501532ed7bcd5bcc6e7f02d80fad2438738d1f47d5b128c0d03928b725917b" Oct 30 17:00:19.226117 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:19.225389 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k_odh-ai-gateway-infra(73890793-133c-4e7b-903e-5a185f8b0711)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" podUID="73890793-133c-4e7b-903e-5a185f8b0711" Oct 30 17:00:19.227978 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.227950 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" event={"ID":"db07007a-80e3-40c7-9385-070440683859","Type":"ContainerStarted","Data":"843165c5aa43bfd8c2eaa77d7a4c208c6de093230ab82bf05281e78468c66dea"} Oct 30 17:00:19.228292 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.228277 6400 scope.go:122] "RemoveContainer" containerID="843165c5aa43bfd8c2eaa77d7a4c208c6de093230ab82bf05281e78468c66dea" Oct 30 17:00:19.228613 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:19.228581 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq_odh-ai-gateway-infra(db07007a-80e3-40c7-9385-070440683859)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" podUID="db07007a-80e3-40c7-9385-070440683859" Oct 30 17:00:19.231031 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.231005 6400 generic.go:356] "Generic (PLEG): container finished" podID="105526c7-7445-4065-b65d-a51b7178d0ed" containerID="dab85ad5ed591c0592db05fb80a215d0bf3b257d18476a8da881c446dcc78948" exitCode=60 Oct 30 17:00:19.231148 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.231082 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" event={"ID":"105526c7-7445-4065-b65d-a51b7178d0ed","Type":"ContainerDied","Data":"dab85ad5ed591c0592db05fb80a215d0bf3b257d18476a8da881c446dcc78948"} Oct 30 17:00:19.231447 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.231428 6400 scope.go:122] "RemoveContainer" containerID="dab85ad5ed591c0592db05fb80a215d0bf3b257d18476a8da881c446dcc78948" Oct 30 17:00:19.231864 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:19.231841 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c_odh-ai-gateway-infra(105526c7-7445-4065-b65d-a51b7178d0ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" Oct 30 17:00:19.234983 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.234961 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" event={"ID":"02eb1598-1e89-4828-bc01-ce68c0b851fb","Type":"ContainerStarted","Data":"46dc2f2f98cb00238d8f87605f550ece0e35740615a13edc520d8b86b410ff25"} Oct 30 17:00:19.235278 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.235263 6400 scope.go:122] "RemoveContainer" containerID="46dc2f2f98cb00238d8f87605f550ece0e35740615a13edc520d8b86b410ff25" Oct 30 17:00:19.235564 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:19.235545 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82_odh-ai-gateway-infra(02eb1598-1e89-4828-bc01-ce68c0b851fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" Oct 30 17:00:19.236265 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.236251 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" Oct 30 17:00:19.236345 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.236291 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m" event={"ID":"2dc73836-c587-4d59-8c93-3b53d3a12ba6","Type":"ContainerDied","Data":"b622dc64e314e97920bce2df5715e38ab28788f747602a8ebc9c663094fb3fb3"} Oct 30 17:00:19.236345 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.236327 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b622dc64e314e97920bce2df5715e38ab28788f747602a8ebc9c663094fb3fb3" Oct 30 17:00:19.276493 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.276461 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h"] Oct 30 17:00:19.326691 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.326657 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c"] Oct 30 17:00:19.372729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.372475 6400 scope.go:122] "RemoveContainer" containerID="06692ae53af065d4bfe423d8e3dd1d010af49d0a661dee1dc4f5a2304939d563" Oct 30 17:00:19.375042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.373956 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2"] Oct 30 17:00:19.400582 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.397193 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k"] Oct 30 17:00:19.464790 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.464763 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" Oct 30 17:00:19.541293 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.541271 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" Oct 30 17:00:19.575344 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.572353 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-maas-api-service-ca\") pod \"8b26fddb-3e85-48f0-af47-ffeb5b512dc2\" (UID: \"8b26fddb-3e85-48f0-af47-ffeb5b512dc2\") " Oct 30 17:00:19.575344 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.572554 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-kube-api-access-mwpts\" (UniqueName: \"kubernetes.io/projected/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-kube-api-access-mwpts\") pod \"8b26fddb-3e85-48f0-af47-ffeb5b512dc2\" (UID: \"8b26fddb-3e85-48f0-af47-ffeb5b512dc2\") " Oct 30 17:00:19.575957 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.575930 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-kube-api-access-mwpts" pod "8b26fddb-3e85-48f0-af47-ffeb5b512dc2" (UID: "8b26fddb-3e85-48f0-af47-ffeb5b512dc2"). InnerVolumeSpecName "kube-api-access-mwpts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:19.578091 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.577940 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-maas-api-service-ca" pod "8b26fddb-3e85-48f0-af47-ffeb5b512dc2" (UID: "8b26fddb-3e85-48f0-af47-ffeb5b512dc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:19.674191 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.674162 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fd855be1-393e-4132-b51a-e8b9e0aed09a-kube-api-access-ccgpc\" (UniqueName: \"kubernetes.io/projected/fd855be1-393e-4132-b51a-e8b9e0aed09a-kube-api-access-ccgpc\") pod \"fd855be1-393e-4132-b51a-e8b9e0aed09a\" (UID: \"fd855be1-393e-4132-b51a-e8b9e0aed09a\") " Oct 30 17:00:19.674391 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.674379 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fd855be1-393e-4132-b51a-e8b9e0aed09a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd855be1-393e-4132-b51a-e8b9e0aed09a-maas-api-service-ca\") pod \"fd855be1-393e-4132-b51a-e8b9e0aed09a\" (UID: \"fd855be1-393e-4132-b51a-e8b9e0aed09a\") " Oct 30 17:00:19.674781 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.674752 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd855be1-393e-4132-b51a-e8b9e0aed09a-maas-api-service-ca" pod "fd855be1-393e-4132-b51a-e8b9e0aed09a" (UID: "fd855be1-393e-4132-b51a-e8b9e0aed09a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:19.674888 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.674865 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:19.674888 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.674885 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mwpts\" (UniqueName: \"kubernetes.io/projected/8b26fddb-3e85-48f0-af47-ffeb5b512dc2-kube-api-access-mwpts\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:19.674962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.674894 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd855be1-393e-4132-b51a-e8b9e0aed09a-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:19.676449 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.676431 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd855be1-393e-4132-b51a-e8b9e0aed09a-kube-api-access-ccgpc" pod "fd855be1-393e-4132-b51a-e8b9e0aed09a" (UID: "fd855be1-393e-4132-b51a-e8b9e0aed09a"). InnerVolumeSpecName "kube-api-access-ccgpc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:19.776019 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:19.775985 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccgpc\" (UniqueName: \"kubernetes.io/projected/fd855be1-393e-4132-b51a-e8b9e0aed09a-kube-api-access-ccgpc\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.250901 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.250866 6400 generic.go:356] "Generic (PLEG): container finished" podID="db07007a-80e3-40c7-9385-070440683859" containerID="843165c5aa43bfd8c2eaa77d7a4c208c6de093230ab82bf05281e78468c66dea" exitCode=60 Oct 30 17:00:20.251382 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.250903 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" event={"ID":"db07007a-80e3-40c7-9385-070440683859","Type":"ContainerDied","Data":"843165c5aa43bfd8c2eaa77d7a4c208c6de093230ab82bf05281e78468c66dea"} Oct 30 17:00:20.251382 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.250986 6400 scope.go:122] "RemoveContainer" containerID="23f625e82a8872587311d1fa85581a5beb3b16f5e67c7a70454be4d3aec6b7e0" Oct 30 17:00:20.251485 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.251435 6400 scope.go:122] "RemoveContainer" containerID="843165c5aa43bfd8c2eaa77d7a4c208c6de093230ab82bf05281e78468c66dea" Oct 30 17:00:20.251916 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:20.251875 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq_odh-ai-gateway-infra(db07007a-80e3-40c7-9385-070440683859)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" podUID="db07007a-80e3-40c7-9385-070440683859" Oct 30 17:00:20.255200 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.255176 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" event={"ID":"fd855be1-393e-4132-b51a-e8b9e0aed09a","Type":"ContainerDied","Data":"db1a478986cce3b2944a448ab40ca0ebd71264cefd8c0ddca528c944e33f174d"} Oct 30 17:00:20.255307 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.255263 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2" Oct 30 17:00:20.261588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.261492 6400 generic.go:356] "Generic (PLEG): container finished" podID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerID="46dc2f2f98cb00238d8f87605f550ece0e35740615a13edc520d8b86b410ff25" exitCode=60 Oct 30 17:00:20.261740 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.261725 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" event={"ID":"02eb1598-1e89-4828-bc01-ce68c0b851fb","Type":"ContainerDied","Data":"46dc2f2f98cb00238d8f87605f550ece0e35740615a13edc520d8b86b410ff25"} Oct 30 17:00:20.262211 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.262190 6400 scope.go:122] "RemoveContainer" containerID="46dc2f2f98cb00238d8f87605f550ece0e35740615a13edc520d8b86b410ff25" Oct 30 17:00:20.262659 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:20.262547 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82_odh-ai-gateway-infra(02eb1598-1e89-4828-bc01-ce68c0b851fb)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" Oct 30 17:00:20.267091 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.267068 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" event={"ID":"8b26fddb-3e85-48f0-af47-ffeb5b512dc2","Type":"ContainerDied","Data":"1a61ec6c7b707f6a2aedebe2d84ed591fdd87ff053c0855901c25cdd78a1fc22"} Oct 30 17:00:20.267226 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.267111 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h" Oct 30 17:00:20.269748 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.269727 6400 scope.go:122] "RemoveContainer" containerID="eb807ccba0495fc1291c1bdd15e9498bae4f8623516b36806cfb11bd64e27fae" Oct 30 17:00:20.283773 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.283746 6400 scope.go:122] "RemoveContainer" containerID="c4eafc770564f550795a1818a462538cdff19b4cc251bcabd08d0802dd5aca32" Oct 30 17:00:20.303723 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.303686 6400 scope.go:122] "RemoveContainer" containerID="8ae927b556c51c6e8b322b4e57a571e9947cfdec743525059ad9aec99d92dbc7" Oct 30 17:00:20.374708 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.373692 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2"] Oct 30 17:00:20.379043 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.379018 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-750dcb-29889660-zp6v2"] Oct 30 17:00:20.415033 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.415006 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq"] Oct 30 17:00:20.446395 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.446364 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" Oct 30 17:00:20.453728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.453544 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h"] Oct 30 17:00:20.459123 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.459090 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-3e2f96-29889660-sv24h"] Oct 30 17:00:20.487556 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.487527 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" Oct 30 17:00:20.492387 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.492365 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82"] Oct 30 17:00:20.587680 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.587576 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/105526c7-7445-4065-b65d-a51b7178d0ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105526c7-7445-4065-b65d-a51b7178d0ed-maas-api-service-ca\") pod \"105526c7-7445-4065-b65d-a51b7178d0ed\" (UID: \"105526c7-7445-4065-b65d-a51b7178d0ed\") " Oct 30 17:00:20.588400 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.588376 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/105526c7-7445-4065-b65d-a51b7178d0ed-kube-api-access-27k8m\" (UniqueName: \"kubernetes.io/projected/105526c7-7445-4065-b65d-a51b7178d0ed-kube-api-access-27k8m\") pod \"105526c7-7445-4065-b65d-a51b7178d0ed\" (UID: \"105526c7-7445-4065-b65d-a51b7178d0ed\") " Oct 30 17:00:20.588900 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.588882 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73890793-133c-4e7b-903e-5a185f8b0711-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73890793-133c-4e7b-903e-5a185f8b0711-maas-api-service-ca\") pod \"73890793-133c-4e7b-903e-5a185f8b0711\" (UID: \"73890793-133c-4e7b-903e-5a185f8b0711\") " Oct 30 17:00:20.589066 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.589051 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73890793-133c-4e7b-903e-5a185f8b0711-kube-api-access-89x8s\" (UniqueName: \"kubernetes.io/projected/73890793-133c-4e7b-903e-5a185f8b0711-kube-api-access-89x8s\") pod \"73890793-133c-4e7b-903e-5a185f8b0711\" (UID: \"73890793-133c-4e7b-903e-5a185f8b0711\") " Oct 30 17:00:20.589685 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.588199 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/105526c7-7445-4065-b65d-a51b7178d0ed-maas-api-service-ca" pod "105526c7-7445-4065-b65d-a51b7178d0ed" (UID: "105526c7-7445-4065-b65d-a51b7178d0ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.589945 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.589926 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/105526c7-7445-4065-b65d-a51b7178d0ed-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.589945 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.589933 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73890793-133c-4e7b-903e-5a185f8b0711-maas-api-service-ca" pod "73890793-133c-4e7b-903e-5a185f8b0711" (UID: "73890793-133c-4e7b-903e-5a185f8b0711"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:20.591574 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.591540 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/105526c7-7445-4065-b65d-a51b7178d0ed-kube-api-access-27k8m" pod "105526c7-7445-4065-b65d-a51b7178d0ed" (UID: "105526c7-7445-4065-b65d-a51b7178d0ed"). InnerVolumeSpecName "kube-api-access-27k8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.592086 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.592067 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73890793-133c-4e7b-903e-5a185f8b0711-kube-api-access-89x8s" pod "73890793-133c-4e7b-903e-5a185f8b0711" (UID: "73890793-133c-4e7b-903e-5a185f8b0711"). InnerVolumeSpecName "kube-api-access-89x8s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:20.691995 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.691953 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89x8s\" (UniqueName: \"kubernetes.io/projected/73890793-133c-4e7b-903e-5a185f8b0711-kube-api-access-89x8s\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.691995 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.691984 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27k8m\" (UniqueName: \"kubernetes.io/projected/105526c7-7445-4065-b65d-a51b7178d0ed-kube-api-access-27k8m\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.691995 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.691998 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73890793-133c-4e7b-903e-5a185f8b0711-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:20.969465 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.969383 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" path="/var/lib/kubelet/pods/8b26fddb-3e85-48f0-af47-ffeb5b512dc2/volumes" Oct 30 17:00:20.969865 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:20.969851 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" path="/var/lib/kubelet/pods/fd855be1-393e-4132-b51a-e8b9e0aed09a/volumes" Oct 30 17:00:21.279495 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.279444 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" event={"ID":"73890793-133c-4e7b-903e-5a185f8b0711","Type":"ContainerDied","Data":"607a010d1f3988e42cd91262c4d492e10014662e9a11121268c1c1ed5b12d92b"} Oct 30 17:00:21.279947 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.279524 6400 scope.go:122] "RemoveContainer" containerID="9b501532ed7bcd5bcc6e7f02d80fad2438738d1f47d5b128c0d03928b725917b" Oct 30 17:00:21.279947 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.279468 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k" Oct 30 17:00:21.284990 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.284952 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" event={"ID":"105526c7-7445-4065-b65d-a51b7178d0ed","Type":"ContainerDied","Data":"6591acd2491b9837b766b78661add286081086ba85455b8e8981cf9ed040bc20"} Oct 30 17:00:21.285092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.285074 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c" Oct 30 17:00:21.295802 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.295788 6400 scope.go:122] "RemoveContainer" containerID="dab85ad5ed591c0592db05fb80a215d0bf3b257d18476a8da881c446dcc78948" Oct 30 17:00:21.342335 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.341989 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k"] Oct 30 17:00:21.346655 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.346390 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-689899-29889660-sdq8k"] Oct 30 17:00:21.461575 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.461528 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c"] Oct 30 17:00:21.467637 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.467595 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-3ed10a-29889660-cb58c"] Oct 30 17:00:21.520147 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.520118 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" Oct 30 17:00:21.523727 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.523704 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" Oct 30 17:00:21.606880 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.606783 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db07007a-80e3-40c7-9385-070440683859-kube-api-access-d6kt5\" (UniqueName: \"kubernetes.io/projected/db07007a-80e3-40c7-9385-070440683859-kube-api-access-d6kt5\") pod \"db07007a-80e3-40c7-9385-070440683859\" (UID: \"db07007a-80e3-40c7-9385-070440683859\") " Oct 30 17:00:21.606880 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.606874 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02eb1598-1e89-4828-bc01-ce68c0b851fb-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02eb1598-1e89-4828-bc01-ce68c0b851fb-maas-api-service-ca\") pod \"02eb1598-1e89-4828-bc01-ce68c0b851fb\" (UID: \"02eb1598-1e89-4828-bc01-ce68c0b851fb\") " Oct 30 17:00:21.607131 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.607107 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db07007a-80e3-40c7-9385-070440683859-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db07007a-80e3-40c7-9385-070440683859-maas-api-service-ca\") pod \"db07007a-80e3-40c7-9385-070440683859\" (UID: \"db07007a-80e3-40c7-9385-070440683859\") " Oct 30 17:00:21.607660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.607457 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02eb1598-1e89-4828-bc01-ce68c0b851fb-kube-api-access-cfxbk\" (UniqueName: \"kubernetes.io/projected/02eb1598-1e89-4828-bc01-ce68c0b851fb-kube-api-access-cfxbk\") pod \"02eb1598-1e89-4828-bc01-ce68c0b851fb\" (UID: \"02eb1598-1e89-4828-bc01-ce68c0b851fb\") " Oct 30 17:00:21.607660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.607618 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db07007a-80e3-40c7-9385-070440683859-maas-api-service-ca" pod "db07007a-80e3-40c7-9385-070440683859" (UID: "db07007a-80e3-40c7-9385-070440683859"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:21.608477 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.608457 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02eb1598-1e89-4828-bc01-ce68c0b851fb-maas-api-service-ca" pod "02eb1598-1e89-4828-bc01-ce68c0b851fb" (UID: "02eb1598-1e89-4828-bc01-ce68c0b851fb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:21.608785 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.608720 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/db07007a-80e3-40c7-9385-070440683859-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.610139 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.609990 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db07007a-80e3-40c7-9385-070440683859-kube-api-access-d6kt5" pod "db07007a-80e3-40c7-9385-070440683859" (UID: "db07007a-80e3-40c7-9385-070440683859"). InnerVolumeSpecName "kube-api-access-d6kt5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.613906 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.613871 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02eb1598-1e89-4828-bc01-ce68c0b851fb-kube-api-access-cfxbk" pod "02eb1598-1e89-4828-bc01-ce68c0b851fb" (UID: "02eb1598-1e89-4828-bc01-ce68c0b851fb"). InnerVolumeSpecName "kube-api-access-cfxbk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:21.710343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.710308 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfxbk\" (UniqueName: \"kubernetes.io/projected/02eb1598-1e89-4828-bc01-ce68c0b851fb-kube-api-access-cfxbk\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.710343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.710339 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6kt5\" (UniqueName: \"kubernetes.io/projected/db07007a-80e3-40c7-9385-070440683859-kube-api-access-d6kt5\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:21.710343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:21.710350 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02eb1598-1e89-4828-bc01-ce68c0b851fb-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:22.299792 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.299758 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" Oct 30 17:00:22.300324 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.299755 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq" event={"ID":"db07007a-80e3-40c7-9385-070440683859","Type":"ContainerDied","Data":"974a1027f28e6ac6d76c7a38e7017624812a7770143fa212d33126252bea0b5b"} Oct 30 17:00:22.300324 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.300035 6400 scope.go:122] "RemoveContainer" containerID="843165c5aa43bfd8c2eaa77d7a4c208c6de093230ab82bf05281e78468c66dea" Oct 30 17:00:22.308752 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.308720 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" event={"ID":"02eb1598-1e89-4828-bc01-ce68c0b851fb","Type":"ContainerDied","Data":"faa93385a80fc1a662fb5857c3804bd5627c28a2863ea7cff8d5ef5ccb33f97c"} Oct 30 17:00:22.308752 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.308743 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82" Oct 30 17:00:22.314160 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.314142 6400 scope.go:122] "RemoveContainer" containerID="46dc2f2f98cb00238d8f87605f550ece0e35740615a13edc520d8b86b410ff25" Oct 30 17:00:22.364931 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.364896 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq"] Oct 30 17:00:22.368934 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.368910 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-1b7bb3-29889660-8m5qq"] Oct 30 17:00:22.401427 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.401395 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82"] Oct 30 17:00:22.409581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.409551 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-9c1e26-29889660-v9j82"] Oct 30 17:00:22.969095 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.969063 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" path="/var/lib/kubelet/pods/02eb1598-1e89-4828-bc01-ce68c0b851fb/volumes" Oct 30 17:00:22.969511 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.969498 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" path="/var/lib/kubelet/pods/105526c7-7445-4065-b65d-a51b7178d0ed/volumes" Oct 30 17:00:22.969905 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.969892 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73890793-133c-4e7b-903e-5a185f8b0711" path="/var/lib/kubelet/pods/73890793-133c-4e7b-903e-5a185f8b0711/volumes" Oct 30 17:00:22.970554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:22.970540 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db07007a-80e3-40c7-9385-070440683859" path="/var/lib/kubelet/pods/db07007a-80e3-40c7-9385-070440683859/volumes" Oct 30 17:00:24.608286 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:24.608233 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:00:39.457623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:39.456724 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965"] Oct 30 17:00:42.244236 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.244154 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc"] Oct 30 17:00:42.244676 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.244421 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" podUID="440019cb-b006-43dc-b146-291a66ec787b" containerName="maas-api" containerID="cri-o://2171dc0597aca21fd700997a235e3a9610b0251a4bcac2161059c97fdeacb327" gracePeriod=30 Oct 30 17:00:42.540149 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.540111 6400 generic.go:356] "Generic (PLEG): container finished" podID="440019cb-b006-43dc-b146-291a66ec787b" containerID="2171dc0597aca21fd700997a235e3a9610b0251a4bcac2161059c97fdeacb327" exitCode=0 Oct 30 17:00:42.540304 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.540276 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" event={"ID":"440019cb-b006-43dc-b146-291a66ec787b","Type":"ContainerDied","Data":"2171dc0597aca21fd700997a235e3a9610b0251a4bcac2161059c97fdeacb327"} Oct 30 17:00:42.579664 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.578043 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 17:00:42.623251 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.623212 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/440019cb-b006-43dc-b146-291a66ec787b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440019cb-b006-43dc-b146-291a66ec787b-openshift-service-ca\") pod \"440019cb-b006-43dc-b146-291a66ec787b\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " Oct 30 17:00:42.623430 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.623275 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-maas-api-tls\") pod \"440019cb-b006-43dc-b146-291a66ec787b\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " Oct 30 17:00:42.623430 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.623348 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/440019cb-b006-43dc-b146-291a66ec787b-kube-api-access-jvptg\" (UniqueName: \"kubernetes.io/projected/440019cb-b006-43dc-b146-291a66ec787b-kube-api-access-jvptg\") pod \"440019cb-b006-43dc-b146-291a66ec787b\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " Oct 30 17:00:42.623430 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.623382 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-metrics-certs\") pod \"440019cb-b006-43dc-b146-291a66ec787b\" (UID: \"440019cb-b006-43dc-b146-291a66ec787b\") " Oct 30 17:00:42.628106 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.628069 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/440019cb-b006-43dc-b146-291a66ec787b-openshift-service-ca" pod "440019cb-b006-43dc-b146-291a66ec787b" (UID: "440019cb-b006-43dc-b146-291a66ec787b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:42.642901 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.633756 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/440019cb-b006-43dc-b146-291a66ec787b-kube-api-access-jvptg" pod "440019cb-b006-43dc-b146-291a66ec787b" (UID: "440019cb-b006-43dc-b146-291a66ec787b"). InnerVolumeSpecName "kube-api-access-jvptg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:42.649432 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.649391 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-metrics-certs" pod "440019cb-b006-43dc-b146-291a66ec787b" (UID: "440019cb-b006-43dc-b146-291a66ec787b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:42.649891 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.649864 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-maas-api-tls" pod "440019cb-b006-43dc-b146-291a66ec787b" (UID: "440019cb-b006-43dc-b146-291a66ec787b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:42.724983 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.724944 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/440019cb-b006-43dc-b146-291a66ec787b-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:42.724983 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.724979 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:42.724983 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.724990 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvptg\" (UniqueName: \"kubernetes.io/projected/440019cb-b006-43dc-b146-291a66ec787b-kube-api-access-jvptg\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:42.725296 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:42.724998 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/440019cb-b006-43dc-b146-291a66ec787b-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:43.424192 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:43.424146 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m"] Oct 30 17:00:43.435241 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:43.435214 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-17f60f-kmz2m"] Oct 30 17:00:43.549816 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:43.549765 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" Oct 30 17:00:43.549816 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:43.549805 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc" event={"ID":"440019cb-b006-43dc-b146-291a66ec787b","Type":"ContainerDied","Data":"c225363ecd1dfba036ab19858a7c14f86b666bd9a414103f77d4ad1b90ca15c1"} Oct 30 17:00:43.550268 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:43.549869 6400 scope.go:122] "RemoveContainer" containerID="2171dc0597aca21fd700997a235e3a9610b0251a4bcac2161059c97fdeacb327" Oct 30 17:00:43.608903 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:43.608864 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc"] Oct 30 17:00:43.616691 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:43.616660 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-17f60f-7557fb4679-hn9qc"] Oct 30 17:00:44.968298 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:44.968263 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc73836-c587-4d59-8c93-3b53d3a12ba6" path="/var/lib/kubelet/pods/2dc73836-c587-4d59-8c93-3b53d3a12ba6/volumes" Oct 30 17:00:44.968854 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:44.968836 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="440019cb-b006-43dc-b146-291a66ec787b" path="/var/lib/kubelet/pods/440019cb-b006-43dc-b146-291a66ec787b/volumes" Oct 30 17:00:46.707031 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:46.706969 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw"] Oct 30 17:00:46.707870 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:46.707278 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" podUID="eabab492-b8ae-4c10-8c3f-d82e63c85ee1" containerName="maas-api" containerID="cri-o://7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7" gracePeriod=30 Oct 30 17:00:47.098285 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.098251 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 17:00:47.269236 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.269198 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-maas-api-tls\") pod \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " Oct 30 17:00:47.269445 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.269309 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-kube-api-access-qfw85\" (UniqueName: \"kubernetes.io/projected/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-kube-api-access-qfw85\") pod \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " Oct 30 17:00:47.269445 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.269362 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-openshift-service-ca\") pod \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " Oct 30 17:00:47.269582 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.269454 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-metrics-certs\") pod \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\" (UID: \"eabab492-b8ae-4c10-8c3f-d82e63c85ee1\") " Oct 30 17:00:47.271097 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.271064 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-openshift-service-ca" pod "eabab492-b8ae-4c10-8c3f-d82e63c85ee1" (UID: "eabab492-b8ae-4c10-8c3f-d82e63c85ee1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:00:47.272693 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.272657 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-kube-api-access-qfw85" pod "eabab492-b8ae-4c10-8c3f-d82e63c85ee1" (UID: "eabab492-b8ae-4c10-8c3f-d82e63c85ee1"). InnerVolumeSpecName "kube-api-access-qfw85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:47.273982 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.273938 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-metrics-certs" pod "eabab492-b8ae-4c10-8c3f-d82e63c85ee1" (UID: "eabab492-b8ae-4c10-8c3f-d82e63c85ee1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:47.274759 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.274415 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-maas-api-tls" pod "eabab492-b8ae-4c10-8c3f-d82e63c85ee1" (UID: "eabab492-b8ae-4c10-8c3f-d82e63c85ee1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:47.370999 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.370919 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:47.370999 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.370950 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:47.370999 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.370960 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfw85\" (UniqueName: \"kubernetes.io/projected/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-kube-api-access-qfw85\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:47.370999 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.370970 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eabab492-b8ae-4c10-8c3f-d82e63c85ee1-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:47.581217 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.581185 6400 generic.go:356] "Generic (PLEG): container finished" podID="eabab492-b8ae-4c10-8c3f-d82e63c85ee1" containerID="7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7" exitCode=0 Oct 30 17:00:47.581433 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.581233 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" event={"ID":"eabab492-b8ae-4c10-8c3f-d82e63c85ee1","Type":"ContainerDied","Data":"7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7"} Oct 30 17:00:47.581433 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.581251 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" Oct 30 17:00:47.581433 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.581282 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw" event={"ID":"eabab492-b8ae-4c10-8c3f-d82e63c85ee1","Type":"ContainerDied","Data":"b47e6416fd16fa30d5d02c2ad1935df72483e93932399342da88f7c03c948444"} Oct 30 17:00:47.581433 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.581320 6400 scope.go:122] "RemoveContainer" containerID="7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7" Oct 30 17:00:47.592817 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.592794 6400 scope.go:122] "RemoveContainer" containerID="7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7" Oct 30 17:00:47.593099 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:47.593081 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7\": container with ID starting with 7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7 not found: ID does not exist" containerID="7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7" Oct 30 17:00:47.593172 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.593110 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7"} err="failed to get container status \"7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7\": rpc error: code = NotFound desc = could not find container \"7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7\": container with ID starting with 7d5a20c92e1bdfe03a733be6f27e307dab1bf53d3716f10d28c83887ef535fa7 not found: ID does not exist" Oct 30 17:00:47.643315 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.643251 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw"] Oct 30 17:00:47.655008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:47.654985 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-9c1e26-5cff47cc96-bg7zw"] Oct 30 17:00:48.968232 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:48.968202 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eabab492-b8ae-4c10-8c3f-d82e63c85ee1" path="/var/lib/kubelet/pods/eabab492-b8ae-4c10-8c3f-d82e63c85ee1/volumes" Oct 30 17:00:54.486523 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.486483 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" podUID="84b45e9d-16ec-4b55-a535-2b9fb9ac2137" containerName="main" containerID="cri-o://b5fd342751fa3d52bd899d6f0b63259bb2ee09bfe0a87eba8e9288fd7be5153c" gracePeriod=45 Oct 30 17:00:54.601633 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:00:54.601575 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:00:54.656291 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.656256 6400 generic.go:356] "Generic (PLEG): container finished" podID="84b45e9d-16ec-4b55-a535-2b9fb9ac2137" containerID="b5fd342751fa3d52bd899d6f0b63259bb2ee09bfe0a87eba8e9288fd7be5153c" exitCode=0 Oct 30 17:00:54.656482 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.656453 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" event={"ID":"84b45e9d-16ec-4b55-a535-2b9fb9ac2137","Type":"ContainerDied","Data":"b5fd342751fa3d52bd899d6f0b63259bb2ee09bfe0a87eba8e9288fd7be5153c"} Oct 30 17:00:54.787349 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.787323 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 17:00:54.888188 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.888155 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-dshm\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-dshm\") pod \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " Oct 30 17:00:54.888394 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.888296 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-model-cache\") pod \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " Oct 30 17:00:54.888394 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.888332 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-home\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-home\") pod \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " Oct 30 17:00:54.888515 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.888402 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-kube-api-access-fzn2t\" (UniqueName: \"kubernetes.io/projected/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-kube-api-access-fzn2t\") pod \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " Oct 30 17:00:54.888676 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.888647 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tls-certs\" (UniqueName: \"kubernetes.io/secret/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tls-certs\") pod \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " Oct 30 17:00:54.888762 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.888687 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-model-cache" pod "84b45e9d-16ec-4b55-a535-2b9fb9ac2137" (UID: "84b45e9d-16ec-4b55-a535-2b9fb9ac2137"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.888806 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.888774 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tmp-dir\") pod \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\" (UID: \"84b45e9d-16ec-4b55-a535-2b9fb9ac2137\") " Oct 30 17:00:54.889059 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.889034 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-home" pod "84b45e9d-16ec-4b55-a535-2b9fb9ac2137" (UID: "84b45e9d-16ec-4b55-a535-2b9fb9ac2137"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.889059 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.889034 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tmp-dir" pod "84b45e9d-16ec-4b55-a535-2b9fb9ac2137" (UID: "84b45e9d-16ec-4b55-a535-2b9fb9ac2137"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.889532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.889483 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.889532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.889516 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.889532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.889529 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.891175 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.891154 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-dshm" pod "84b45e9d-16ec-4b55-a535-2b9fb9ac2137" (UID: "84b45e9d-16ec-4b55-a535-2b9fb9ac2137"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:00:54.891507 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.891483 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-kube-api-access-fzn2t" pod "84b45e9d-16ec-4b55-a535-2b9fb9ac2137" (UID: "84b45e9d-16ec-4b55-a535-2b9fb9ac2137"). InnerVolumeSpecName "kube-api-access-fzn2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:00:54.891802 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.891773 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tls-certs" pod "84b45e9d-16ec-4b55-a535-2b9fb9ac2137" (UID: "84b45e9d-16ec-4b55-a535-2b9fb9ac2137"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:00:54.990461 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.990426 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fzn2t\" (UniqueName: \"kubernetes.io/projected/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-kube-api-access-fzn2t\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.990461 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.990459 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:54.990626 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:54.990473 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84b45e9d-16ec-4b55-a535-2b9fb9ac2137-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:00:55.676627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:55.673698 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" Oct 30 17:00:55.676627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:55.673726 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" event={"ID":"84b45e9d-16ec-4b55-a535-2b9fb9ac2137","Type":"ContainerDied","Data":"0074fd02adbb40b5fae83c9d33874940ef2f14dcd541d345d2a877bef22b3c80"} Oct 30 17:00:55.676627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:55.673774 6400 scope.go:122] "RemoveContainer" containerID="b5fd342751fa3d52bd899d6f0b63259bb2ee09bfe0a87eba8e9288fd7be5153c" Oct 30 17:00:55.746039 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:55.745994 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965"] Oct 30 17:00:55.747524 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:55.747493 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965" podUID="84b45e9d-16ec-4b55-a535-2b9fb9ac2137" containerName="main" probeResult="failure" output="Get \"https://10.134.0.130:8000/ready\": context deadline exceeded" Oct 30 17:00:55.751359 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:55.751325 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/auth-test-model-3ed10a-kserve-7b6854478f-zd965"] Oct 30 17:00:56.968659 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:00:56.968629 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b45e9d-16ec-4b55-a535-2b9fb9ac2137" path="/var/lib/kubelet/pods/84b45e9d-16ec-4b55-a535-2b9fb9ac2137/volumes" Oct 30 17:01:02.667876 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.667845 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j"] Oct 30 17:01:02.668394 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668381 6400 cpu_manager.go:414] "containerMap: removing container" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerName="cleanup" Oct 30 17:01:02.668444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668395 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerName="cleanup" Oct 30 17:01:02.668444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668406 6400 cpu_manager.go:414] "containerMap: removing container" podUID="73890793-133c-4e7b-903e-5a185f8b0711" containerName="cleanup" Oct 30 17:01:02.668444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668411 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73890793-133c-4e7b-903e-5a185f8b0711" containerName="cleanup" Oct 30 17:01:02.668444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668420 6400 cpu_manager.go:414] "containerMap: removing container" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerName="cleanup" Oct 30 17:01:02.668444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668426 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerName="cleanup" Oct 30 17:01:02.668444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668438 6400 cpu_manager.go:414] "containerMap: removing container" podUID="db07007a-80e3-40c7-9385-070440683859" containerName="cleanup" Oct 30 17:01:02.668444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668442 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db07007a-80e3-40c7-9385-070440683859" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668451 6400 cpu_manager.go:414] "containerMap: removing container" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668456 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668467 6400 cpu_manager.go:414] "containerMap: removing container" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668472 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668477 6400 cpu_manager.go:414] "containerMap: removing container" podUID="84b45e9d-16ec-4b55-a535-2b9fb9ac2137" containerName="main" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668481 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b45e9d-16ec-4b55-a535-2b9fb9ac2137" containerName="main" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668490 6400 cpu_manager.go:414] "containerMap: removing container" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668494 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668501 6400 cpu_manager.go:414] "containerMap: removing container" podUID="eabab492-b8ae-4c10-8c3f-d82e63c85ee1" containerName="maas-api" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668505 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eabab492-b8ae-4c10-8c3f-d82e63c85ee1" containerName="maas-api" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668512 6400 cpu_manager.go:414] "containerMap: removing container" podUID="2dc73836-c587-4d59-8c93-3b53d3a12ba6" containerName="revoke-keys" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668519 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc73836-c587-4d59-8c93-3b53d3a12ba6" containerName="revoke-keys" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668533 6400 cpu_manager.go:414] "containerMap: removing container" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668538 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668544 6400 cpu_manager.go:414] "containerMap: removing container" podUID="73890793-133c-4e7b-903e-5a185f8b0711" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668548 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73890793-133c-4e7b-903e-5a185f8b0711" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668560 6400 cpu_manager.go:414] "containerMap: removing container" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668565 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668570 6400 cpu_manager.go:414] "containerMap: removing container" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668574 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668585 6400 cpu_manager.go:414] "containerMap: removing container" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668589 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668596 6400 cpu_manager.go:414] "containerMap: removing container" podUID="440019cb-b006-43dc-b146-291a66ec787b" containerName="maas-api" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668616 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="440019cb-b006-43dc-b146-291a66ec787b" containerName="maas-api" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668628 6400 cpu_manager.go:414] "containerMap: removing container" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668636 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668648 6400 cpu_manager.go:414] "containerMap: removing container" podUID="db07007a-80e3-40c7-9385-070440683859" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668653 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db07007a-80e3-40c7-9385-070440683859" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668659 6400 cpu_manager.go:414] "containerMap: removing container" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668663 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668764 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="eabab492-b8ae-4c10-8c3f-d82e63c85ee1" containerName="maas-api" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668772 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668778 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc73836-c587-4d59-8c93-3b53d3a12ba6" containerName="revoke-keys" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668785 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668792 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="73890793-133c-4e7b-903e-5a185f8b0711" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668799 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668804 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668809 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668815 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668819 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerName="cleanup" Oct 30 17:01:02.668796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668826 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="440019cb-b006-43dc-b146-291a66ec787b" containerName="maas-api" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668830 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerName="cleanup" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668836 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="db07007a-80e3-40c7-9385-070440683859" containerName="cleanup" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668842 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="db07007a-80e3-40c7-9385-070440683859" containerName="cleanup" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668847 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerName="cleanup" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668852 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerName="cleanup" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668857 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="73890793-133c-4e7b-903e-5a185f8b0711" containerName="cleanup" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668862 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerName="cleanup" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668868 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="73890793-133c-4e7b-903e-5a185f8b0711" containerName="cleanup" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668874 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerName="cleanup" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668887 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="db07007a-80e3-40c7-9385-070440683859" containerName="cleanup" Oct 30 17:01:02.670588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.668892 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b45e9d-16ec-4b55-a535-2b9fb9ac2137" containerName="main" Oct 30 17:01:02.673768 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.673750 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:01:02.699719 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.699691 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-b4214a\"" Oct 30 17:01:02.711011 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.710983 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-b4214a-dockercfg-qqsr8\"" Oct 30 17:01:02.714721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.714696 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j"] Oct 30 17:01:02.762934 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.762413 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq"] Oct 30 17:01:02.764081 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.764054 6400 cpu_manager.go:414] "containerMap: removing container" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" containerName="cleanup" Oct 30 17:01:02.764230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.764216 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" containerName="cleanup" Oct 30 17:01:02.764362 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.764350 6400 cpu_manager.go:414] "containerMap: removing container" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerName="cleanup" Oct 30 17:01:02.764440 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.764429 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerName="cleanup" Oct 30 17:01:02.764529 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.764519 6400 cpu_manager.go:414] "containerMap: removing container" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerName="cleanup" Oct 30 17:01:02.764690 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.764675 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b26fddb-3e85-48f0-af47-ffeb5b512dc2" containerName="cleanup" Oct 30 17:01:02.764825 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.764814 6400 cpu_manager.go:414] "containerMap: removing container" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerName="cleanup" Oct 30 17:01:02.764923 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.764911 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fd855be1-393e-4132-b51a-e8b9e0aed09a" containerName="cleanup" Oct 30 17:01:02.765008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.764998 6400 cpu_manager.go:414] "containerMap: removing container" podUID="73890793-133c-4e7b-903e-5a185f8b0711" containerName="cleanup" Oct 30 17:01:02.765085 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.765075 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73890793-133c-4e7b-903e-5a185f8b0711" containerName="cleanup" Oct 30 17:01:02.765171 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.765162 6400 cpu_manager.go:414] "containerMap: removing container" podUID="db07007a-80e3-40c7-9385-070440683859" containerName="cleanup" Oct 30 17:01:02.765248 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.765238 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db07007a-80e3-40c7-9385-070440683859" containerName="cleanup" Oct 30 17:01:02.765491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.765477 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf131cbf-5e5a-4e53-b630-5bb585d90398" containerName="cleanup" Oct 30 17:01:02.765614 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.765586 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="105526c7-7445-4065-b65d-a51b7178d0ed" containerName="cleanup" Oct 30 17:01:02.765720 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.765709 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="02eb1598-1e89-4828-bc01-ce68c0b851fb" containerName="cleanup" Oct 30 17:01:02.814381 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.814352 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq"] Oct 30 17:01:02.814769 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.814704 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:01:02.873577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.860406 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f72c6\" (UniqueName: \"kubernetes.io/projected/52ba393f-b73e-4e8e-b509-12dabb21e851-kube-api-access-f72c6\") pod \"payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq\" (UID: \"52ba393f-b73e-4e8e-b509-12dabb21e851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:01:02.873577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.860496 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ba393f-b73e-4e8e-b509-12dabb21e851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq\" (UID: \"52ba393f-b73e-4e8e-b509-12dabb21e851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:01:02.873577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.860531 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j\" (UID: \"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:01:02.873577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.860635 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75995\" (UniqueName: \"kubernetes.io/projected/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-kube-api-access-75995\") pod \"payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j\" (UID: \"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:01:02.962143 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.962051 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75995\" (UniqueName: \"kubernetes.io/projected/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-kube-api-access-75995\") pod \"payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j\" (UID: \"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:01:02.962319 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.962220 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f72c6\" (UniqueName: \"kubernetes.io/projected/52ba393f-b73e-4e8e-b509-12dabb21e851-kube-api-access-f72c6\") pod \"payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq\" (UID: \"52ba393f-b73e-4e8e-b509-12dabb21e851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:01:02.962319 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.962289 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ba393f-b73e-4e8e-b509-12dabb21e851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq\" (UID: \"52ba393f-b73e-4e8e-b509-12dabb21e851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:01:02.962461 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.962328 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j\" (UID: \"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:01:02.963036 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.963011 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j\" (UID: \"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:01:02.963219 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.963196 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ba393f-b73e-4e8e-b509-12dabb21e851-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq\" (UID: \"52ba393f-b73e-4e8e-b509-12dabb21e851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:01:02.986328 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.986294 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75995\" (UniqueName: \"kubernetes.io/projected/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-kube-api-access-75995\") pod \"payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j\" (UID: \"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:01:02.986683 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.986665 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f72c6\" (UniqueName: \"kubernetes.io/projected/52ba393f-b73e-4e8e-b509-12dabb21e851-kube-api-access-f72c6\") pod \"payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq\" (UID: \"52ba393f-b73e-4e8e-b509-12dabb21e851\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:01:02.987442 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:02.987427 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:01:03.211980 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:03.211948 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:01:03.343704 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:03.343665 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j"] Oct 30 17:01:03.527981 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:03.527955 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq"] Oct 30 17:01:03.528056 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:01:03.527954 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52ba393f_b73e_4e8e_b509_12dabb21e851.slice/crio-2c648b950dc0f0a7594fabaafd6f147e402ad5b188fff0b91138f0a227e7a0ce WatchSource:0}: Error finding container 2c648b950dc0f0a7594fabaafd6f147e402ad5b188fff0b91138f0a227e7a0ce: Status 404 returned error can't find the container with id 2c648b950dc0f0a7594fabaafd6f147e402ad5b188fff0b91138f0a227e7a0ce Oct 30 17:01:03.750627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:03.749840 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" event={"ID":"52ba393f-b73e-4e8e-b509-12dabb21e851","Type":"ContainerStarted","Data":"2c648b950dc0f0a7594fabaafd6f147e402ad5b188fff0b91138f0a227e7a0ce"} Oct 30 17:01:03.758622 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:03.751591 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" event={"ID":"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d","Type":"ContainerStarted","Data":"9570c435ad3a0d6907589b18c27547a74a885f4ef7b32fd783d84bc713c334bc"} Oct 30 17:01:06.782321 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:06.782286 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" event={"ID":"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d","Type":"ContainerStarted","Data":"d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3"} Oct 30 17:01:06.783932 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:06.783906 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" event={"ID":"52ba393f-b73e-4e8e-b509-12dabb21e851","Type":"ContainerStarted","Data":"dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353"} Oct 30 17:01:06.784073 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:06.783966 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:01:06.825983 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:06.825918 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" podStartSLOduration=2.4119030329999998 podStartE2EDuration="4.825901404s" totalImagesPullingTime="2.413998371s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:06.821457593 +0000 UTC m=+2375.971738591" watchObservedRunningTime="2026-10-30 17:01:06.825901404 +0000 UTC m=+2375.976182393" Oct 30 17:01:06.869388 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:06.869336 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" podStartSLOduration=2.529956256 podStartE2EDuration="4.869320358s" totalImagesPullingTime="2.339364102s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:06.867280888 +0000 UTC m=+2376.017561878" watchObservedRunningTime="2026-10-30 17:01:06.869320358 +0000 UTC m=+2376.019601379" Oct 30 17:01:07.791020 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:07.790985 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:01:13.316128 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.316088 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7"] Oct 30 17:01:13.321905 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.321876 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.340778 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.340750 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7"] Oct 30 17:01:13.347060 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.347029 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3ed10a\"/\"openshift-service-ca.crt\"" Oct 30 17:01:13.350707 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.350681 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w4-3ed10a\"/\"kube-root-ca.crt\"" Oct 30 17:01:13.350839 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.350780 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3ed10a\"/\"rate-test-model-3ed10a-kserve-self-signed-certs\"" Oct 30 17:01:13.350839 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.350834 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w4-3ed10a\"/\"default-dockercfg-jg82r\"" Oct 30 17:01:13.424201 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.424166 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-model-cache\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.424393 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.424225 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-dshm\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.424452 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.424382 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-home\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.424452 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.424413 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pxd7h\" (UniqueName: \"kubernetes.io/projected/0fd9b015-c869-45c1-9482-fc0e46acf69d-kube-api-access-pxd7h\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.424558 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.424450 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd9b015-c869-45c1-9482-fc0e46acf69d-tls-certs\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.424620 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.424563 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-tmp-dir\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.525491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.525449 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-model-cache\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.525741 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.525501 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-dshm\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.525741 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.525634 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-home\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.525741 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.525664 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pxd7h\" (UniqueName: \"kubernetes.io/projected/0fd9b015-c869-45c1-9482-fc0e46acf69d-kube-api-access-pxd7h\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.525741 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.525697 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd9b015-c869-45c1-9482-fc0e46acf69d-tls-certs\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.525741 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.525741 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-tmp-dir\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.526045 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.526021 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-home\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.526119 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.526099 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-tmp-dir\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.526248 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.526226 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-model-cache\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.528866 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.528826 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd9b015-c869-45c1-9482-fc0e46acf69d-tls-certs\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.529071 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.529052 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-dshm\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.564362 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.564320 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pxd7h\" (UniqueName: \"kubernetes.io/projected/0fd9b015-c869-45c1-9482-fc0e46acf69d-kube-api-access-pxd7h\") pod \"rate-test-model-3ed10a-kserve-7bb48d77-8fvh7\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.636672 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.635821 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:13.965464 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:13.965433 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7"] Oct 30 17:01:13.965464 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:01:13.965453 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0fd9b015_c869_45c1_9482_fc0e46acf69d.slice/crio-9ea0660afe91ce904fd51240cb47586b776604473e0c07b2afcbdfc53ec8c977 WatchSource:0}: Error finding container 9ea0660afe91ce904fd51240cb47586b776604473e0c07b2afcbdfc53ec8c977: Status 404 returned error can't find the container with id 9ea0660afe91ce904fd51240cb47586b776604473e0c07b2afcbdfc53ec8c977 Oct 30 17:01:14.812383 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.812354 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r"] Oct 30 17:01:14.816709 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.816687 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:14.845906 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.845874 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-05810b\"/\"sub-test-model-05810b-kserve-self-signed-certs\"" Oct 30 17:01:14.846113 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.846017 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-05810b\"/\"default-dockercfg-8qx8t\"" Oct 30 17:01:14.846113 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.846078 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-05810b\"/\"openshift-service-ca.crt\"" Oct 30 17:01:14.846113 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.846077 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" event={"ID":"0fd9b015-c869-45c1-9482-fc0e46acf69d","Type":"ContainerStarted","Data":"567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d"} Oct 30 17:01:14.846302 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.846123 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" event={"ID":"0fd9b015-c869-45c1-9482-fc0e46acf69d","Type":"ContainerStarted","Data":"9ea0660afe91ce904fd51240cb47586b776604473e0c07b2afcbdfc53ec8c977"} Oct 30 17:01:14.846302 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.846132 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-05810b\"/\"kube-root-ca.crt\"" Oct 30 17:01:14.850029 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.850006 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r"] Oct 30 17:01:14.902597 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.902533 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" podStartSLOduration=1.902514985 podStartE2EDuration="1.902514985s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:14.897897084 +0000 UTC m=+2384.048178073" watchObservedRunningTime="2026-10-30 17:01:14.902514985 +0000 UTC m=+2384.052795975" Oct 30 17:01:14.949061 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.949024 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hqvxv\" (UniqueName: \"kubernetes.io/projected/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-kube-api-access-hqvxv\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:14.949263 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.949206 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-home\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:14.949339 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.949275 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tls-certs\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:14.949434 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.949404 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-model-cache\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:14.949576 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.949527 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tmp-dir\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:14.949673 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:14.949590 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-dshm\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.051291 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.051249 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tmp-dir\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.051525 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.051352 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-dshm\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.052087 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.051736 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tmp-dir\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.052087 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.051755 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hqvxv\" (UniqueName: \"kubernetes.io/projected/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-kube-api-access-hqvxv\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.052087 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.051855 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-home\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.052087 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.051992 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tls-certs\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.052395 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.052369 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-home\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.053227 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.052502 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-model-cache\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.053227 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.052889 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-model-cache\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.059149 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.058044 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-dshm\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.059149 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.058810 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tls-certs\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.097684 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.091620 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hqvxv\" (UniqueName: \"kubernetes.io/projected/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-kube-api-access-hqvxv\") pod \"sub-test-model-05810b-kserve-57dc5f5b85-4ww8r\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.138653 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.138618 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:15.477255 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.477228 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r"] Oct 30 17:01:15.478791 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:01:15.478763 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1ddbbe4a_21cf_4281_94ce_490b0469e0ce.slice/crio-bb162acb016968bed08bdcd6e87987bb052105e131d301d065e9550c692b4641 WatchSource:0}: Error finding container bb162acb016968bed08bdcd6e87987bb052105e131d301d065e9550c692b4641: Status 404 returned error can't find the container with id bb162acb016968bed08bdcd6e87987bb052105e131d301d065e9550c692b4641 Oct 30 17:01:15.640687 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.640656 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:15.855577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.855533 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" event={"ID":"1ddbbe4a-21cf-4281-94ce-490b0469e0ce","Type":"ContainerStarted","Data":"31f95bf97210552264db07ad249d13e4606c5aa5354f96cd52fb81475bf38d36"} Oct 30 17:01:15.855577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.855578 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" event={"ID":"1ddbbe4a-21cf-4281-94ce-490b0469e0ce","Type":"ContainerStarted","Data":"bb162acb016968bed08bdcd6e87987bb052105e131d301d065e9550c692b4641"} Oct 30 17:01:15.898418 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:15.898360 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" podStartSLOduration=1.898347297 podStartE2EDuration="1.898347297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:15.895050845 +0000 UTC m=+2385.045331832" watchObservedRunningTime="2026-10-30 17:01:15.898347297 +0000 UTC m=+2385.048628285" Oct 30 17:01:16.139991 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:16.139914 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:17.794320 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:17.794287 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:01:18.801041 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:18.801015 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:01:18.898012 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:18.897977 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx"] Oct 30 17:01:18.903431 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:18.903407 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:18.927644 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:18.924050 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-bfecdd\"/\"openshift-service-ca.crt\"" Oct 30 17:01:18.927644 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:18.926743 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-bfecdd\"/\"default-dockercfg-d6v5d\"" Oct 30 17:01:18.932219 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:18.928367 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-bfecdd\"/\"kube-root-ca.crt\"" Oct 30 17:01:18.932219 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:18.929184 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-bfecdd\"/\"auth-test-model-bfecdd-kserve-self-signed-certs\"" Oct 30 17:01:18.938095 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:18.938064 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx"] Oct 30 17:01:19.009011 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.008976 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-home\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.009011 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.009025 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5wl7\" (UniqueName: \"kubernetes.io/projected/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-kube-api-access-q5wl7\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.009286 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.009165 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tls-certs\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.009286 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.009278 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tmp-dir\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.009407 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.009333 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-model-cache\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.009470 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.009437 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-dshm\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.111162 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.111071 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-home\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.111162 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.111125 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5wl7\" (UniqueName: \"kubernetes.io/projected/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-kube-api-access-q5wl7\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.111406 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.111195 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tls-certs\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.111406 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.111297 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tmp-dir\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.111597 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.111553 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-model-cache\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.111728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.111576 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-home\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.111790 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.111735 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tmp-dir\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.112061 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.112039 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-model-cache\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.112420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.112401 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-dshm\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.114445 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.114402 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tls-certs\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.115434 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.115411 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-dshm\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.132102 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.132077 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5wl7\" (UniqueName: \"kubernetes.io/projected/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-kube-api-access-q5wl7\") pod \"auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.244829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.244535 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:19.559039 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.559007 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx"] Oct 30 17:01:19.560211 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:01:19.560179 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod33e1f0a0_69e7_4cfc_b76f_ca88e55c3d48.slice/crio-e1e0d891a6603e359909800d95ae2cfb9a0589f079aacb840add163f659238d4 WatchSource:0}: Error finding container e1e0d891a6603e359909800d95ae2cfb9a0589f079aacb840add163f659238d4: Status 404 returned error can't find the container with id e1e0d891a6603e359909800d95ae2cfb9a0589f079aacb840add163f659238d4 Oct 30 17:01:19.886852 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.886769 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" event={"ID":"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48","Type":"ContainerStarted","Data":"2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267"} Oct 30 17:01:19.886852 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.886808 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" event={"ID":"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48","Type":"ContainerStarted","Data":"e1e0d891a6603e359909800d95ae2cfb9a0589f079aacb840add163f659238d4"} Oct 30 17:01:19.925059 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:19.925006 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" podStartSLOduration=1.924992007 podStartE2EDuration="1.924992007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:19.923134821 +0000 UTC m=+2389.073415811" watchObservedRunningTime="2026-10-30 17:01:19.924992007 +0000 UTC m=+2389.075272996" Oct 30 17:01:20.251295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:20.246735 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:23.640390 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:23.640355 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:23.653285 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:23.653260 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:23.930703 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:23.930595 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:01:24.600981 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:01:24.600936 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:01:25.139739 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:25.139709 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:25.152534 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:25.152509 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:25.945867 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:25.945841 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:01:29.245350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:29.245306 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:29.258185 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:29.258158 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:29.980674 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:29.980649 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:01:42.729080 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:42.729036 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft"] Oct 30 17:01:42.739003 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:42.729263 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" podUID="31009489-863e-4dad-9818-d662fa62d429" containerName="maas-api" containerID="cri-o://a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73" gracePeriod=30 Oct 30 17:01:43.042065 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.042037 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 17:01:43.060183 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.060153 6400 generic.go:356] "Generic (PLEG): container finished" podID="31009489-863e-4dad-9818-d662fa62d429" containerID="a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73" exitCode=0 Oct 30 17:01:43.060368 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.060189 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" event={"ID":"31009489-863e-4dad-9818-d662fa62d429","Type":"ContainerDied","Data":"a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73"} Oct 30 17:01:43.060368 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.060224 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" Oct 30 17:01:43.060368 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.060258 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft" event={"ID":"31009489-863e-4dad-9818-d662fa62d429","Type":"ContainerDied","Data":"b7df83b33f33d4c0ac3885d9e7e51d7acdafaba446dfbaa42534d3aae3f35149"} Oct 30 17:01:43.060368 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.060286 6400 scope.go:122] "RemoveContainer" containerID="a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73" Oct 30 17:01:43.072652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.072625 6400 scope.go:122] "RemoveContainer" containerID="a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73" Oct 30 17:01:43.072990 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:01:43.072959 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73\": container with ID starting with a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73 not found: ID does not exist" containerID="a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73" Oct 30 17:01:43.073113 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.073000 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73"} err="failed to get container status \"a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73\": rpc error: code = NotFound desc = could not find container \"a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73\": container with ID starting with a2151507b741e1102176103a593c0c06658e1fd21113d0491881b9a949204e73 not found: ID does not exist" Oct 30 17:01:43.084272 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.084247 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31009489-863e-4dad-9818-d662fa62d429-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31009489-863e-4dad-9818-d662fa62d429-openshift-service-ca\") pod \"31009489-863e-4dad-9818-d662fa62d429\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " Oct 30 17:01:43.084722 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.084686 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31009489-863e-4dad-9818-d662fa62d429-kube-api-access-k6np5\" (UniqueName: \"kubernetes.io/projected/31009489-863e-4dad-9818-d662fa62d429-kube-api-access-k6np5\") pod \"31009489-863e-4dad-9818-d662fa62d429\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " Oct 30 17:01:43.084957 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.084939 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-metrics-certs\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-metrics-certs\") pod \"31009489-863e-4dad-9818-d662fa62d429\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " Oct 30 17:01:43.085179 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.085002 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31009489-863e-4dad-9818-d662fa62d429-openshift-service-ca" pod "31009489-863e-4dad-9818-d662fa62d429" (UID: "31009489-863e-4dad-9818-d662fa62d429"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:43.085291 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.085274 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-maas-api-tls\") pod \"31009489-863e-4dad-9818-d662fa62d429\" (UID: \"31009489-863e-4dad-9818-d662fa62d429\") " Oct 30 17:01:43.086902 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.086875 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/31009489-863e-4dad-9818-d662fa62d429-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:43.087732 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.087695 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31009489-863e-4dad-9818-d662fa62d429-kube-api-access-k6np5" pod "31009489-863e-4dad-9818-d662fa62d429" (UID: "31009489-863e-4dad-9818-d662fa62d429"). InnerVolumeSpecName "kube-api-access-k6np5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:43.087732 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.087697 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-metrics-certs" pod "31009489-863e-4dad-9818-d662fa62d429" (UID: "31009489-863e-4dad-9818-d662fa62d429"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:43.087973 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.087939 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-maas-api-tls" pod "31009489-863e-4dad-9818-d662fa62d429" (UID: "31009489-863e-4dad-9818-d662fa62d429"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:01:43.188157 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.188112 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6np5\" (UniqueName: \"kubernetes.io/projected/31009489-863e-4dad-9818-d662fa62d429-kube-api-access-k6np5\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:43.188157 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.188154 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:43.188425 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.188169 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/31009489-863e-4dad-9818-d662fa62d429-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:43.288751 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.288664 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8959-424b98"] Oct 30 17:01:43.289881 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.289852 6400 cpu_manager.go:414] "containerMap: removing container" podUID="31009489-863e-4dad-9818-d662fa62d429" containerName="maas-api" Oct 30 17:01:43.290052 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.290038 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31009489-863e-4dad-9818-d662fa62d429" containerName="maas-api" Oct 30 17:01:43.290383 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.290367 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="31009489-863e-4dad-9818-d662fa62d429" containerName="maas-api" Oct 30 17:01:43.305929 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.305885 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8959-424b98" Oct 30 17:01:43.345489 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.345410 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr4c7\"" Oct 30 17:01:43.378497 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.376811 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8959-424b98"] Oct 30 17:01:43.397409 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.397358 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbth6\" (UniqueName: \"kubernetes.io/projected/4005f382-6ec0-484b-97d2-912a0e2964b4-kube-api-access-nbth6\") pod \"test-curl-8959-424b98\" (UID: \"4005f382-6ec0-484b-97d2-912a0e2964b4\") " pod="opendatahub/test-curl-8959-424b98" Oct 30 17:01:43.427320 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.427290 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft"] Oct 30 17:01:43.430465 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.430441 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-689899-55ccb9d775-xjqft"] Oct 30 17:01:43.499040 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.499003 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbth6\" (UniqueName: \"kubernetes.io/projected/4005f382-6ec0-484b-97d2-912a0e2964b4-kube-api-access-nbth6\") pod \"test-curl-8959-424b98\" (UID: \"4005f382-6ec0-484b-97d2-912a0e2964b4\") " pod="opendatahub/test-curl-8959-424b98" Oct 30 17:01:43.521001 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.520969 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbth6\" (UniqueName: \"kubernetes.io/projected/4005f382-6ec0-484b-97d2-912a0e2964b4-kube-api-access-nbth6\") pod \"test-curl-8959-424b98\" (UID: \"4005f382-6ec0-484b-97d2-912a0e2964b4\") " pod="opendatahub/test-curl-8959-424b98" Oct 30 17:01:43.688325 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.688250 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8959-424b98" Oct 30 17:01:43.989815 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:43.989780 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8959-424b98"] Oct 30 17:01:43.990923 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:01:43.990896 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4005f382_6ec0_484b_97d2_912a0e2964b4.slice/crio-527c044c0ddd5d53d8c8b2dc1ec1de272e9f9ea597a4c6a07b14e1b11acc541f WatchSource:0}: Error finding container 527c044c0ddd5d53d8c8b2dc1ec1de272e9f9ea597a4c6a07b14e1b11acc541f: Status 404 returned error can't find the container with id 527c044c0ddd5d53d8c8b2dc1ec1de272e9f9ea597a4c6a07b14e1b11acc541f Oct 30 17:01:44.070892 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:44.070858 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8959-424b98" event={"ID":"4005f382-6ec0-484b-97d2-912a0e2964b4","Type":"ContainerStarted","Data":"7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51"} Oct 30 17:01:44.071029 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:44.070906 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8959-424b98" event={"ID":"4005f382-6ec0-484b-97d2-912a0e2964b4","Type":"ContainerStarted","Data":"527c044c0ddd5d53d8c8b2dc1ec1de272e9f9ea597a4c6a07b14e1b11acc541f"} Oct 30 17:01:44.110510 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:44.110399 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8959-424b98" podStartSLOduration=1.11038317 podStartE2EDuration="1.11038317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:44.109522364 +0000 UTC m=+2413.259803352" watchObservedRunningTime="2026-10-30 17:01:44.11038317 +0000 UTC m=+2413.260664158" Oct 30 17:01:44.968924 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:44.968891 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31009489-863e-4dad-9818-d662fa62d429" path="/var/lib/kubelet/pods/31009489-863e-4dad-9818-d662fa62d429/volumes" Oct 30 17:01:45.084046 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:45.084016 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8959-424b98"] Oct 30 17:01:45.089459 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:45.089430 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8959-424b98"] Oct 30 17:01:46.087540 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.087473 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8959-424b98" podUID="4005f382-6ec0-484b-97d2-912a0e2964b4" containerName="test-curl-8959-424b98" containerID="cri-o://7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51" gracePeriod=2 Oct 30 17:01:46.096045 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.096019 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn"] Oct 30 17:01:46.096871 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.096849 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4005f382-6ec0-484b-97d2-912a0e2964b4" containerName="test-curl-8959-424b98" Oct 30 17:01:46.096871 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.096870 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4005f382-6ec0-484b-97d2-912a0e2964b4" containerName="test-curl-8959-424b98" Oct 30 17:01:46.097042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.097031 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4005f382-6ec0-484b-97d2-912a0e2964b4" containerName="test-curl-8959-424b98" Oct 30 17:01:46.101170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.101151 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.101706 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:01:46.101681 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8959-424b98\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="4005f382-6ec0-484b-97d2-912a0e2964b4" pod="opendatahub/test-curl-8959-424b98" Oct 30 17:01:46.115295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.115275 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c4b84\"/\"kube-root-ca.crt\"" Oct 30 17:01:46.115394 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.115362 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c4b84\"/\"openshift-service-ca.crt\"" Oct 30 17:01:46.128403 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.128381 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn"] Oct 30 17:01:46.139080 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.139042 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47bmx\" (UniqueName: \"kubernetes.io/projected/93da5f58-2c4f-4753-b177-26a7f86d8eb4-kube-api-access-47bmx\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.139237 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.139219 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-dshm\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.139322 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.139279 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tmp-dir\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.139442 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.139415 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tls-certs\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.139534 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.139514 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-home\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.139672 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.139650 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-model-cache\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.147099 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:01:46.147066 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8959-424b98\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="4005f382-6ec0-484b-97d2-912a0e2964b4" pod="opendatahub/test-curl-8959-424b98" Oct 30 17:01:46.200187 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.200151 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c4b84\"/\"rate-test-model-7c4b84-kserve-self-signed-certs\"" Oct 30 17:01:46.200187 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.200179 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w4-7c4b84\"/\"default-dockercfg-kjprt\"" Oct 30 17:01:46.241361 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.241324 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tls-certs\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.241536 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.241375 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-home\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.241536 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.241451 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-model-cache\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.241536 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.241511 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47bmx\" (UniqueName: \"kubernetes.io/projected/93da5f58-2c4f-4753-b177-26a7f86d8eb4-kube-api-access-47bmx\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.241771 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.241637 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-dshm\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.241771 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.241687 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tmp-dir\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.241890 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.241869 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-home\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.242076 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.242054 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tmp-dir\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.242207 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.242180 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-model-cache\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.244711 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.244689 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-dshm\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.247313 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.247283 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tls-certs\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.264788 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.264756 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-47bmx\" (UniqueName: \"kubernetes.io/projected/93da5f58-2c4f-4753-b177-26a7f86d8eb4-kube-api-access-47bmx\") pod \"rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.417675 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.416962 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:46.733984 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:46.733956 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn"] Oct 30 17:01:46.734957 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:01:46.734933 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93da5f58_2c4f_4753_b177_26a7f86d8eb4.slice/crio-b3fa6a5fb3a6d12f9723ad6c4e8daa7c491f9c458e75cc2919949ef2fb40a2f6 WatchSource:0}: Error finding container b3fa6a5fb3a6d12f9723ad6c4e8daa7c491f9c458e75cc2919949ef2fb40a2f6: Status 404 returned error can't find the container with id b3fa6a5fb3a6d12f9723ad6c4e8daa7c491f9c458e75cc2919949ef2fb40a2f6 Oct 30 17:01:47.096835 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:47.096792 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" event={"ID":"93da5f58-2c4f-4753-b177-26a7f86d8eb4","Type":"ContainerStarted","Data":"22bb7915341b5d298cb18f298c2b5789e2e1c650ea584515044e162d07b3ca2d"} Oct 30 17:01:47.097232 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:47.096845 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" event={"ID":"93da5f58-2c4f-4753-b177-26a7f86d8eb4","Type":"ContainerStarted","Data":"b3fa6a5fb3a6d12f9723ad6c4e8daa7c491f9c458e75cc2919949ef2fb40a2f6"} Oct 30 17:01:47.142043 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:47.141963 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" podStartSLOduration=1.141937591 podStartE2EDuration="1.141937591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:01:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:01:47.14186389 +0000 UTC m=+2416.292144878" watchObservedRunningTime="2026-10-30 17:01:47.141937591 +0000 UTC m=+2416.292218656" Oct 30 17:01:47.417957 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:47.417869 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:48.301808 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:48.301782 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8959-424b98" Oct 30 17:01:48.380928 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:48.380840 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4005f382-6ec0-484b-97d2-912a0e2964b4-kube-api-access-nbth6\" (UniqueName: \"kubernetes.io/projected/4005f382-6ec0-484b-97d2-912a0e2964b4-kube-api-access-nbth6\") pod \"4005f382-6ec0-484b-97d2-912a0e2964b4\" (UID: \"4005f382-6ec0-484b-97d2-912a0e2964b4\") " Oct 30 17:01:48.383214 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:48.383184 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4005f382-6ec0-484b-97d2-912a0e2964b4-kube-api-access-nbth6" pod "4005f382-6ec0-484b-97d2-912a0e2964b4" (UID: "4005f382-6ec0-484b-97d2-912a0e2964b4"). InnerVolumeSpecName "kube-api-access-nbth6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:48.483089 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:48.483050 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbth6\" (UniqueName: \"kubernetes.io/projected/4005f382-6ec0-484b-97d2-912a0e2964b4-kube-api-access-nbth6\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:48.968415 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:48.968379 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4005f382-6ec0-484b-97d2-912a0e2964b4" path="/var/lib/kubelet/pods/4005f382-6ec0-484b-97d2-912a0e2964b4/volumes" Oct 30 17:01:49.113072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:49.113031 6400 generic.go:356] "Generic (PLEG): container finished" podID="4005f382-6ec0-484b-97d2-912a0e2964b4" containerID="7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51" exitCode=137 Oct 30 17:01:49.113271 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:49.113075 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8959-424b98" Oct 30 17:01:49.113271 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:49.113116 6400 scope.go:122] "RemoveContainer" containerID="7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51" Oct 30 17:01:49.123878 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:49.123844 6400 scope.go:122] "RemoveContainer" containerID="7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51" Oct 30 17:01:49.124156 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:01:49.124135 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51\": container with ID starting with 7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51 not found: ID does not exist" containerID="7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51" Oct 30 17:01:49.124233 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:49.124163 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51"} err="failed to get container status \"7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51\": rpc error: code = NotFound desc = could not find container \"7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51\": container with ID starting with 7dfc3e195e52532b951a5e8d28fc04de432cb5e22e6077aebc9115bdda508c51 not found: ID does not exist" Oct 30 17:01:50.995221 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:50.995187 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb"] Oct 30 17:01:50.995943 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:50.995471 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerName="istio-proxy" containerID="cri-o://3e7e4515450bbf17a0ee727edeae6d391da7f285ee24cfbff79b5d4b706d67a0" gracePeriod=30 Oct 30 17:01:50.996368 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:50.996325 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp"] Oct 30 17:01:50.997235 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:50.996564 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerName="istio-proxy" containerID="cri-o://0276e06a14430bc2e2d665d10a63aef20feb233d546582242a58cb529f1a4d90" gracePeriod=30 Oct 30 17:01:51.025491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:51.025446 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p"] Oct 30 17:01:51.025992 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:51.025937 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerName="istio-proxy" containerID="cri-o://f4f6cf3d9bf600fc393f65f4e7c1db637b9c9e88784d9db31945d9ca63770c40" gracePeriod=30 Oct 30 17:01:54.599860 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:01:54.599824 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:01:56.136146 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.136117 6400 patch_prober.go:29] interesting pod/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body= Oct 30 17:01:56.136571 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.136180 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" Oct 30 17:01:56.166807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.166512 6400 generic.go:356] "Generic (PLEG): container finished" podID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerID="f4f6cf3d9bf600fc393f65f4e7c1db637b9c9e88784d9db31945d9ca63770c40" exitCode=0 Oct 30 17:01:56.166807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.166667 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" event={"ID":"b38e5e4e-341a-458f-93f1-771f464ddb8d","Type":"ContainerDied","Data":"f4f6cf3d9bf600fc393f65f4e7c1db637b9c9e88784d9db31945d9ca63770c40"} Oct 30 17:01:56.168899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.168851 6400 generic.go:356] "Generic (PLEG): container finished" podID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerID="0276e06a14430bc2e2d665d10a63aef20feb233d546582242a58cb529f1a4d90" exitCode=0 Oct 30 17:01:56.169068 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.168976 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" event={"ID":"07688b22-fe9c-49b1-9436-bd8f642cae88","Type":"ContainerDied","Data":"0276e06a14430bc2e2d665d10a63aef20feb233d546582242a58cb529f1a4d90"} Oct 30 17:01:56.171037 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.171011 6400 generic.go:356] "Generic (PLEG): container finished" podID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerID="3e7e4515450bbf17a0ee727edeae6d391da7f285ee24cfbff79b5d4b706d67a0" exitCode=0 Oct 30 17:01:56.171152 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.171140 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" event={"ID":"b7761f76-bf36-446a-a128-74ac9fa1a1da","Type":"ContainerDied","Data":"3e7e4515450bbf17a0ee727edeae6d391da7f285ee24cfbff79b5d4b706d67a0"} Oct 30 17:01:56.345035 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.345008 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 17:01:56.353920 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.353898 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 17:01:56.381724 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.381643 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 17:01:56.422350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.420332 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:56.434809 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.434778 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:56.497622 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.497574 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-credential-socket\") pod \"b38e5e4e-341a-458f-93f1-771f464ddb8d\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " Oct 30 17:01:56.497869 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.497851 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07688b22-fe9c-49b1-9436-bd8f642cae88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07688b22-fe9c-49b1-9436-bd8f642cae88-istiod-ca-cert\") pod \"07688b22-fe9c-49b1-9436-bd8f642cae88\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " Oct 30 17:01:56.498037 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498021 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-certs\") pod \"b7761f76-bf36-446a-a128-74ac9fa1a1da\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " Oct 30 17:01:56.498188 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498172 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-podinfo\") pod \"07688b22-fe9c-49b1-9436-bd8f642cae88\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " Oct 30 17:01:56.498311 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498296 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-credential-socket\") pod \"b7761f76-bf36-446a-a128-74ac9fa1a1da\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " Oct 30 17:01:56.498450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498436 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-envoy\") pod \"07688b22-fe9c-49b1-9436-bd8f642cae88\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " Oct 30 17:01:56.498617 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498582 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-socket\") pod \"b7761f76-bf36-446a-a128-74ac9fa1a1da\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " Oct 30 17:01:56.498747 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498732 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-certs\") pod \"b38e5e4e-341a-458f-93f1-771f464ddb8d\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " Oct 30 17:01:56.498893 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498877 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38e5e4e-341a-458f-93f1-771f464ddb8d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38e5e4e-341a-458f-93f1-771f464ddb8d-istiod-ca-cert\") pod \"b38e5e4e-341a-458f-93f1-771f464ddb8d\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " Oct 30 17:01:56.499007 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498993 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-data\") pod \"b7761f76-bf36-446a-a128-74ac9fa1a1da\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " Oct 30 17:01:56.499127 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.499112 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-podinfo\") pod \"b7761f76-bf36-446a-a128-74ac9fa1a1da\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " Oct 30 17:01:56.499237 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.499223 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-envoy\") pod \"b38e5e4e-341a-458f-93f1-771f464ddb8d\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " Oct 30 17:01:56.499383 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.499366 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-token\" (UniqueName: \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-token\") pod \"b7761f76-bf36-446a-a128-74ac9fa1a1da\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " Oct 30 17:01:56.499511 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.499496 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-envoy\") pod \"b7761f76-bf36-446a-a128-74ac9fa1a1da\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " Oct 30 17:01:56.499890 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.499825 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-certs\") pod \"07688b22-fe9c-49b1-9436-bd8f642cae88\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " Oct 30 17:01:56.500069 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.500050 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-kube-api-access-7nlkw\" (UniqueName: \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-kube-api-access-7nlkw\") pod \"b7761f76-bf36-446a-a128-74ac9fa1a1da\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " Oct 30 17:01:56.500211 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.500195 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-socket\") pod \"b38e5e4e-341a-458f-93f1-771f464ddb8d\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " Oct 30 17:01:56.500404 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.500386 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-credential-socket\") pod \"07688b22-fe9c-49b1-9436-bd8f642cae88\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " Oct 30 17:01:56.500550 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.500536 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-kube-api-access-7cr9n\" (UniqueName: \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-kube-api-access-7cr9n\") pod \"b38e5e4e-341a-458f-93f1-771f464ddb8d\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " Oct 30 17:01:56.500705 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.500689 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-data\") pod \"b38e5e4e-341a-458f-93f1-771f464ddb8d\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " Oct 30 17:01:56.500924 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.500906 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7761f76-bf36-446a-a128-74ac9fa1a1da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7761f76-bf36-446a-a128-74ac9fa1a1da-istiod-ca-cert\") pod \"b7761f76-bf36-446a-a128-74ac9fa1a1da\" (UID: \"b7761f76-bf36-446a-a128-74ac9fa1a1da\") " Oct 30 17:01:56.501053 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.501038 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-token\" (UniqueName: \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-token\") pod \"07688b22-fe9c-49b1-9436-bd8f642cae88\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " Oct 30 17:01:56.501212 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.501197 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-kube-api-access-shqfj\" (UniqueName: \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-kube-api-access-shqfj\") pod \"07688b22-fe9c-49b1-9436-bd8f642cae88\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " Oct 30 17:01:56.501342 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.501326 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-socket\") pod \"07688b22-fe9c-49b1-9436-bd8f642cae88\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " Oct 30 17:01:56.501470 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.501447 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-podinfo\") pod \"b38e5e4e-341a-458f-93f1-771f464ddb8d\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " Oct 30 17:01:56.501571 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.501551 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-data\") pod \"07688b22-fe9c-49b1-9436-bd8f642cae88\" (UID: \"07688b22-fe9c-49b1-9436-bd8f642cae88\") " Oct 30 17:01:56.501663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.501582 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-token\" (UniqueName: \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-token\") pod \"b38e5e4e-341a-458f-93f1-771f464ddb8d\" (UID: \"b38e5e4e-341a-458f-93f1-771f464ddb8d\") " Oct 30 17:01:56.503371 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498200 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-credential-socket" pod "b38e5e4e-341a-458f-93f1-771f464ddb8d" (UID: "b38e5e4e-341a-458f-93f1-771f464ddb8d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.503480 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498553 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07688b22-fe9c-49b1-9436-bd8f642cae88-istiod-ca-cert" pod "07688b22-fe9c-49b1-9436-bd8f642cae88" (UID: "07688b22-fe9c-49b1-9436-bd8f642cae88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:56.503480 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.498881 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-certs" pod "b7761f76-bf36-446a-a128-74ac9fa1a1da" (UID: "b7761f76-bf36-446a-a128-74ac9fa1a1da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.503480 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.502053 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-podinfo" pod "07688b22-fe9c-49b1-9436-bd8f642cae88" (UID: "07688b22-fe9c-49b1-9436-bd8f642cae88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:56.503480 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.502198 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-certs" pod "07688b22-fe9c-49b1-9436-bd8f642cae88" (UID: "07688b22-fe9c-49b1-9436-bd8f642cae88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.503480 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.502337 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-credential-socket" pod "b7761f76-bf36-446a-a128-74ac9fa1a1da" (UID: "b7761f76-bf36-446a-a128-74ac9fa1a1da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.503480 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.503174 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-data" pod "b7761f76-bf36-446a-a128-74ac9fa1a1da" (UID: "b7761f76-bf36-446a-a128-74ac9fa1a1da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.503480 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.503345 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-socket" pod "b7761f76-bf36-446a-a128-74ac9fa1a1da" (UID: "b7761f76-bf36-446a-a128-74ac9fa1a1da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.503943 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.503583 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-certs" pod "b38e5e4e-341a-458f-93f1-771f464ddb8d" (UID: "b38e5e4e-341a-458f-93f1-771f464ddb8d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.503943 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.503881 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38e5e4e-341a-458f-93f1-771f464ddb8d-istiod-ca-cert" pod "b38e5e4e-341a-458f-93f1-771f464ddb8d" (UID: "b38e5e4e-341a-458f-93f1-771f464ddb8d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:56.504305 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.504283 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7761f76-bf36-446a-a128-74ac9fa1a1da-istiod-ca-cert" pod "b7761f76-bf36-446a-a128-74ac9fa1a1da" (UID: "b7761f76-bf36-446a-a128-74ac9fa1a1da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:01:56.505016 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.504988 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-envoy" pod "07688b22-fe9c-49b1-9436-bd8f642cae88" (UID: "07688b22-fe9c-49b1-9436-bd8f642cae88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.506581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.506555 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-envoy" pod "b38e5e4e-341a-458f-93f1-771f464ddb8d" (UID: "b38e5e4e-341a-458f-93f1-771f464ddb8d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.506849 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.506830 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-socket" pod "07688b22-fe9c-49b1-9436-bd8f642cae88" (UID: "07688b22-fe9c-49b1-9436-bd8f642cae88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.507941 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.507919 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-token" pod "07688b22-fe9c-49b1-9436-bd8f642cae88" (UID: "07688b22-fe9c-49b1-9436-bd8f642cae88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.508715 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.508688 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-podinfo" pod "b7761f76-bf36-446a-a128-74ac9fa1a1da" (UID: "b7761f76-bf36-446a-a128-74ac9fa1a1da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:56.509121 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.509100 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-credential-socket" pod "07688b22-fe9c-49b1-9436-bd8f642cae88" (UID: "07688b22-fe9c-49b1-9436-bd8f642cae88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.509461 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.509441 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-socket" pod "b38e5e4e-341a-458f-93f1-771f464ddb8d" (UID: "b38e5e4e-341a-458f-93f1-771f464ddb8d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.509590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.509565 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-kube-api-access-shqfj" pod "07688b22-fe9c-49b1-9436-bd8f642cae88" (UID: "07688b22-fe9c-49b1-9436-bd8f642cae88"). InnerVolumeSpecName "kube-api-access-shqfj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.509846 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.509823 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-data" pod "07688b22-fe9c-49b1-9436-bd8f642cae88" (UID: "07688b22-fe9c-49b1-9436-bd8f642cae88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.511692 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.511563 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-data" pod "b38e5e4e-341a-458f-93f1-771f464ddb8d" (UID: "b38e5e4e-341a-458f-93f1-771f464ddb8d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.511980 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.511812 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-token" pod "b7761f76-bf36-446a-a128-74ac9fa1a1da" (UID: "b7761f76-bf36-446a-a128-74ac9fa1a1da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.512780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.512749 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-podinfo" pod "b38e5e4e-341a-458f-93f1-771f464ddb8d" (UID: "b38e5e4e-341a-458f-93f1-771f464ddb8d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:01:56.513702 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.513555 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-envoy" pod "b7761f76-bf36-446a-a128-74ac9fa1a1da" (UID: "b7761f76-bf36-446a-a128-74ac9fa1a1da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:01:56.519289 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.519265 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-token" pod "b38e5e4e-341a-458f-93f1-771f464ddb8d" (UID: "b38e5e4e-341a-458f-93f1-771f464ddb8d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.524352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.521456 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-kube-api-access-7cr9n" pod "b38e5e4e-341a-458f-93f1-771f464ddb8d" (UID: "b38e5e4e-341a-458f-93f1-771f464ddb8d"). InnerVolumeSpecName "kube-api-access-7cr9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.526757 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.526723 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-kube-api-access-7nlkw" pod "b7761f76-bf36-446a-a128-74ac9fa1a1da" (UID: "b7761f76-bf36-446a-a128-74ac9fa1a1da"). InnerVolumeSpecName "kube-api-access-7nlkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:01:56.603481 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603393 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cr9n\" (UniqueName: \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-kube-api-access-7cr9n\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603481 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603433 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603481 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603448 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b7761f76-bf36-446a-a128-74ac9fa1a1da-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603481 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603461 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603481 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603474 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqfj\" (UniqueName: \"kubernetes.io/projected/07688b22-fe9c-49b1-9436-bd8f642cae88-kube-api-access-shqfj\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603488 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603502 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603514 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603527 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603539 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603552 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07688b22-fe9c-49b1-9436-bd8f642cae88-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603564 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603576 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603589 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603640 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603654 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603667 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603679 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b38e5e4e-341a-458f-93f1-771f464ddb8d-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603691 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603703 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603714 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603721 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603730 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b7761f76-bf36-446a-a128-74ac9fa1a1da-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603737 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603745 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nlkw\" (UniqueName: \"kubernetes.io/projected/b7761f76-bf36-446a-a128-74ac9fa1a1da-kube-api-access-7nlkw\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603755 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b38e5e4e-341a-458f-93f1-771f464ddb8d-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:56.603910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:56.603769 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07688b22-fe9c-49b1-9436-bd8f642cae88-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:01:57.180560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.180528 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" Oct 30 17:01:57.181041 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.180555 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb" event={"ID":"b7761f76-bf36-446a-a128-74ac9fa1a1da","Type":"ContainerDied","Data":"f21a0f253915798e90198f91b694bf06895c0da497d14e63fec9474e49c3df0c"} Oct 30 17:01:57.181041 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.180655 6400 scope.go:122] "RemoveContainer" containerID="3e7e4515450bbf17a0ee727edeae6d391da7f285ee24cfbff79b5d4b706d67a0" Oct 30 17:01:57.182190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.182168 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" Oct 30 17:01:57.182190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.182182 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p" event={"ID":"b38e5e4e-341a-458f-93f1-771f464ddb8d","Type":"ContainerDied","Data":"c70612df2dee9c1a8afe28107ab2481296f767b877d7df8de8b2af3599db1805"} Oct 30 17:01:57.184078 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.184056 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" event={"ID":"07688b22-fe9c-49b1-9436-bd8f642cae88","Type":"ContainerDied","Data":"b2274beff607643ebb529ef8c9bd1ae6b9c75c0c0c54d01c4bae9be0a8aa9f3c"} Oct 30 17:01:57.184206 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.184170 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp" Oct 30 17:01:57.193751 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.193731 6400 scope.go:122] "RemoveContainer" containerID="f4f6cf3d9bf600fc393f65f4e7c1db637b9c9e88784d9db31945d9ca63770c40" Oct 30 17:01:57.200345 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.200326 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:01:57.208515 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.208500 6400 scope.go:122] "RemoveContainer" containerID="0276e06a14430bc2e2d665d10a63aef20feb233d546582242a58cb529f1a4d90" Oct 30 17:01:57.245342 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.245315 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp"] Oct 30 17:01:57.256032 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.256001 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-vnkqp"] Oct 30 17:01:57.287000 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.286973 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb"] Oct 30 17:01:57.295957 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.295937 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-rdwnb"] Oct 30 17:01:57.330657 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.330635 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p"] Oct 30 17:01:57.333989 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:57.333966 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-689899-openshift-default-66cbd45f99-n7k7p"] Oct 30 17:01:58.968800 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:58.968771 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" path="/var/lib/kubelet/pods/07688b22-fe9c-49b1-9436-bd8f642cae88/volumes" Oct 30 17:01:58.969292 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:58.969274 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" path="/var/lib/kubelet/pods/b38e5e4e-341a-458f-93f1-771f464ddb8d/volumes" Oct 30 17:01:58.969884 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:01:58.969869 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" path="/var/lib/kubelet/pods/b7761f76-bf36-446a-a128-74ac9fa1a1da/volumes" Oct 30 17:02:24.603132 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:02:24.603092 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:02:29.418502 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:29.417967 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h"] Oct 30 17:02:31.122767 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.122736 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q"] Oct 30 17:02:31.123260 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.123247 6400 cpu_manager.go:414] "containerMap: removing container" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerName="istio-proxy" Oct 30 17:02:31.123308 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.123261 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerName="istio-proxy" Oct 30 17:02:31.123308 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.123293 6400 cpu_manager.go:414] "containerMap: removing container" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerName="istio-proxy" Oct 30 17:02:31.123308 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.123298 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerName="istio-proxy" Oct 30 17:02:31.123407 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.123309 6400 cpu_manager.go:414] "containerMap: removing container" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerName="istio-proxy" Oct 30 17:02:31.123407 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.123315 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerName="istio-proxy" Oct 30 17:02:31.123475 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.123410 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="07688b22-fe9c-49b1-9436-bd8f642cae88" containerName="istio-proxy" Oct 30 17:02:31.123475 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.123422 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7761f76-bf36-446a-a128-74ac9fa1a1da" containerName="istio-proxy" Oct 30 17:02:31.123475 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.123429 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38e5e4e-341a-458f-93f1-771f464ddb8d" containerName="istio-proxy" Oct 30 17:02:31.127793 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.127774 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.158491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.158466 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-3ed10a-openshift-default-dockercfg-gt5hj\"" Oct 30 17:02:31.159280 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.159255 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q"] Oct 30 17:02:31.196776 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.196734 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-data\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.196955 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.196795 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-certs\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.197022 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.196956 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brvgv\" (UniqueName: \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-kube-api-access-brvgv\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.197095 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.197017 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-podinfo\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.197095 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.197058 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istiod-ca-cert\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.197212 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.197089 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-envoy\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.197212 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.197181 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-token\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.197326 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.197241 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.197413 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.197385 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-credential-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.298574 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.298539 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-credential-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.298783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.298625 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-data\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.298783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.298653 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-certs\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.298783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.298673 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brvgv\" (UniqueName: \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-kube-api-access-brvgv\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.298783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.298695 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-podinfo\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.298783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.298722 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istiod-ca-cert\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.298783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.298754 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-envoy\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.299108 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.298838 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-token\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.299108 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.298898 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.299218 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.299174 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-certs\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.299386 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.299361 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-credential-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.299472 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.299435 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-socket\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.299545 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.299487 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istiod-ca-cert\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.299545 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.299528 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-data\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.301296 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.301259 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-envoy\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.301497 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.301482 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-podinfo\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.321032 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.321007 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-token\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.321206 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.321089 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brvgv\" (UniqueName: \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-kube-api-access-brvgv\") pod \"e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.443703 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.443593 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:31.784398 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.784373 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q"] Oct 30 17:02:31.785301 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:02:31.785272 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1e4f957_c66c_40dc_8560_a6bbd2cd75d1.slice/crio-bdb5d6abfd2b69828e756a9ca0a803fee5f03848ee8b28b3667259bc53b998c4 WatchSource:0}: Error finding container bdb5d6abfd2b69828e756a9ca0a803fee5f03848ee8b28b3667259bc53b998c4: Status 404 returned error can't find the container with id bdb5d6abfd2b69828e756a9ca0a803fee5f03848ee8b28b3667259bc53b998c4 Oct 30 17:02:31.788046 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.788013 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:02:31.788111 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.788077 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:02:31.788155 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:31.788120 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:02:32.460367 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:32.460329 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" event={"ID":"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1","Type":"ContainerStarted","Data":"6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb"} Oct 30 17:02:32.460806 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:32.460389 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" event={"ID":"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1","Type":"ContainerStarted","Data":"bdb5d6abfd2b69828e756a9ca0a803fee5f03848ee8b28b3667259bc53b998c4"} Oct 30 17:02:32.537740 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:32.537670 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" podStartSLOduration=1.5376475379999999 podStartE2EDuration="1.537647538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:02:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:02:32.519095852 +0000 UTC m=+2461.669376841" watchObservedRunningTime="2026-10-30 17:02:32.537647538 +0000 UTC m=+2461.687928527" Oct 30 17:02:33.446982 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:33.446949 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:33.449211 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:33.449181 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 17:02:33.449426 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:33.449402 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 17:02:34.455775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:34.455738 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 17:02:34.464054 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:34.464011 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 17:02:35.443985 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:35.443943 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 17:02:35.444177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:35.444006 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 17:02:36.444099 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:36.444069 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 17:02:36.444648 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:36.444580 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 17:02:37.443894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:37.443859 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" start-of-body= Oct 30 17:02:37.444082 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:37.443920 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.151:15021/healthz/ready\": dial tcp 10.134.0.151:15021: connect: connection refused" Oct 30 17:02:38.450337 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:38.450303 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:38.450921 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:38.450900 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:38.451551 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:38.451526 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:02:44.452013 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.451955 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" podUID="28e0f55b-622b-4ecf-bb99-5b080c872f4c" containerName="main" containerID="cri-o://7d7e25a31d355788ca4ba1858ed81fcb5b6e29b5b74c8b524229e439d9facf37" gracePeriod=45 Oct 30 17:02:44.598831 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.598792 6400 generic.go:356] "Generic (PLEG): container finished" podID="28e0f55b-622b-4ecf-bb99-5b080c872f4c" containerID="7d7e25a31d355788ca4ba1858ed81fcb5b6e29b5b74c8b524229e439d9facf37" exitCode=0 Oct 30 17:02:44.598986 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.598861 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" event={"ID":"28e0f55b-622b-4ecf-bb99-5b080c872f4c","Type":"ContainerDied","Data":"7d7e25a31d355788ca4ba1858ed81fcb5b6e29b5b74c8b524229e439d9facf37"} Oct 30 17:02:44.730309 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.730287 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 17:02:44.848055 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.848017 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tmp-dir\") pod \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " Oct 30 17:02:44.848261 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.848067 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-dshm\") pod \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " Oct 30 17:02:44.848261 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.848099 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-model-cache\") pod \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " Oct 30 17:02:44.848261 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.848147 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tls-certs\" (UniqueName: \"kubernetes.io/secret/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tls-certs\") pod \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " Oct 30 17:02:44.848460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.848320 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tmp-dir" pod "28e0f55b-622b-4ecf-bb99-5b080c872f4c" (UID: "28e0f55b-622b-4ecf-bb99-5b080c872f4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:44.848460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.848338 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-home\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-home\") pod \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " Oct 30 17:02:44.848460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.848425 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-model-cache" pod "28e0f55b-622b-4ecf-bb99-5b080c872f4c" (UID: "28e0f55b-622b-4ecf-bb99-5b080c872f4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:44.848460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.848453 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28e0f55b-622b-4ecf-bb99-5b080c872f4c-kube-api-access-ls928\" (UniqueName: \"kubernetes.io/projected/28e0f55b-622b-4ecf-bb99-5b080c872f4c-kube-api-access-ls928\") pod \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\" (UID: \"28e0f55b-622b-4ecf-bb99-5b080c872f4c\") " Oct 30 17:02:44.848715 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.848596 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-home" pod "28e0f55b-622b-4ecf-bb99-5b080c872f4c" (UID: "28e0f55b-622b-4ecf-bb99-5b080c872f4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:44.849353 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.849325 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:02:44.849457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.849357 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:02:44.849457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.849371 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:02:44.850571 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.850541 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-dshm" pod "28e0f55b-622b-4ecf-bb99-5b080c872f4c" (UID: "28e0f55b-622b-4ecf-bb99-5b080c872f4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:02:44.850701 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.850567 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tls-certs" pod "28e0f55b-622b-4ecf-bb99-5b080c872f4c" (UID: "28e0f55b-622b-4ecf-bb99-5b080c872f4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:02:44.850768 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.850733 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28e0f55b-622b-4ecf-bb99-5b080c872f4c-kube-api-access-ls928" pod "28e0f55b-622b-4ecf-bb99-5b080c872f4c" (UID: "28e0f55b-622b-4ecf-bb99-5b080c872f4c"). InnerVolumeSpecName "kube-api-access-ls928". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:02:44.950328 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.950301 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ls928\" (UniqueName: \"kubernetes.io/projected/28e0f55b-622b-4ecf-bb99-5b080c872f4c-kube-api-access-ls928\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:02:44.950328 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.950327 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28e0f55b-622b-4ecf-bb99-5b080c872f4c-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:02:44.950502 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:44.950337 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28e0f55b-622b-4ecf-bb99-5b080c872f4c-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:02:45.606845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:45.606817 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" Oct 30 17:02:45.607271 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:45.606861 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h" event={"ID":"28e0f55b-622b-4ecf-bb99-5b080c872f4c","Type":"ContainerDied","Data":"2be94f6a72f6c7da296b212e49e592ff586eaa9d1e734d4354d3f36f8b324460"} Oct 30 17:02:45.607271 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:45.606916 6400 scope.go:122] "RemoveContainer" containerID="7d7e25a31d355788ca4ba1858ed81fcb5b6e29b5b74c8b524229e439d9facf37" Oct 30 17:02:45.654595 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:45.654561 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h"] Oct 30 17:02:45.663934 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:45.663910 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/auth-test-model-dc74d5-kserve-565d7557c-jf44h"] Oct 30 17:02:45.768833 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:45.768803 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz"] Oct 30 17:02:46.970698 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:02:46.970663 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28e0f55b-622b-4ecf-bb99-5b080c872f4c" path="/var/lib/kubelet/pods/28e0f55b-622b-4ecf-bb99-5b080c872f4c/volumes" Oct 30 17:02:54.601559 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:02:54.601523 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:03:00.792331 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:00.792287 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" podUID="81b60d6e-bef0-499e-9cb5-7c935fc0587e" containerName="main" containerID="cri-o://0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa" gracePeriod=45 Oct 30 17:03:01.068336 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.068312 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 17:03:01.192576 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.192549 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tls-certs\" (UniqueName: \"kubernetes.io/secret/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tls-certs\") pod \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " Oct 30 17:03:01.192752 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.192721 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81b60d6e-bef0-499e-9cb5-7c935fc0587e-kube-api-access-p9zdk\" (UniqueName: \"kubernetes.io/projected/81b60d6e-bef0-499e-9cb5-7c935fc0587e-kube-api-access-p9zdk\") pod \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " Oct 30 17:03:01.192941 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.192921 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-home\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-home\") pod \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " Oct 30 17:03:01.192988 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.192964 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-model-cache\") pod \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " Oct 30 17:03:01.193052 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.192985 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-dshm\") pod \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " Oct 30 17:03:01.193052 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.193030 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tmp-dir\") pod \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\" (UID: \"81b60d6e-bef0-499e-9cb5-7c935fc0587e\") " Oct 30 17:03:01.193237 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.193211 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-home" pod "81b60d6e-bef0-499e-9cb5-7c935fc0587e" (UID: "81b60d6e-bef0-499e-9cb5-7c935fc0587e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:01.193322 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.193233 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-model-cache" pod "81b60d6e-bef0-499e-9cb5-7c935fc0587e" (UID: "81b60d6e-bef0-499e-9cb5-7c935fc0587e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:01.193503 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.193404 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tmp-dir" pod "81b60d6e-bef0-499e-9cb5-7c935fc0587e" (UID: "81b60d6e-bef0-499e-9cb5-7c935fc0587e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:01.193759 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.193733 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:03:01.193759 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.193760 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:03:01.193959 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.193773 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:03:01.195170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.195147 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tls-certs" pod "81b60d6e-bef0-499e-9cb5-7c935fc0587e" (UID: "81b60d6e-bef0-499e-9cb5-7c935fc0587e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:03:01.195267 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.195201 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81b60d6e-bef0-499e-9cb5-7c935fc0587e-kube-api-access-p9zdk" pod "81b60d6e-bef0-499e-9cb5-7c935fc0587e" (UID: "81b60d6e-bef0-499e-9cb5-7c935fc0587e"). InnerVolumeSpecName "kube-api-access-p9zdk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:03:01.195385 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.195366 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-dshm" pod "81b60d6e-bef0-499e-9cb5-7c935fc0587e" (UID: "81b60d6e-bef0-499e-9cb5-7c935fc0587e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:03:01.295311 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.295283 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/81b60d6e-bef0-499e-9cb5-7c935fc0587e-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:03:01.295311 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.295309 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9zdk\" (UniqueName: \"kubernetes.io/projected/81b60d6e-bef0-499e-9cb5-7c935fc0587e-kube-api-access-p9zdk\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:03:01.295496 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.295319 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/81b60d6e-bef0-499e-9cb5-7c935fc0587e-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:03:01.719920 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.719885 6400 generic.go:356] "Generic (PLEG): container finished" podID="81b60d6e-bef0-499e-9cb5-7c935fc0587e" containerID="0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa" exitCode=0 Oct 30 17:03:01.720119 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.719934 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" Oct 30 17:03:01.720119 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.719943 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" event={"ID":"81b60d6e-bef0-499e-9cb5-7c935fc0587e","Type":"ContainerDied","Data":"0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa"} Oct 30 17:03:01.720119 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.720004 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz" event={"ID":"81b60d6e-bef0-499e-9cb5-7c935fc0587e","Type":"ContainerDied","Data":"9871c33b0a7f9e71cff30c9848792f0a3786d839cae2f87809fec1af052eda0b"} Oct 30 17:03:01.720119 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.720036 6400 scope.go:122] "RemoveContainer" containerID="0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa" Oct 30 17:03:01.731038 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.731018 6400 scope.go:122] "RemoveContainer" containerID="0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa" Oct 30 17:03:01.731419 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:03:01.731394 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa\": container with ID starting with 0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa not found: ID does not exist" containerID="0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa" Oct 30 17:03:01.731514 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.731431 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa"} err="failed to get container status \"0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa\": rpc error: code = NotFound desc = could not find container \"0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa\": container with ID starting with 0f5c0d0cce5d958fcac3d2bf56a7dc14f9c15802cc262f7d7041ddc0613670fa not found: ID does not exist" Oct 30 17:03:01.765045 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.765021 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz"] Oct 30 17:03:01.775135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:01.775112 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-661509/auth-test-model-661509-kserve-5f4c8f5cb5-2ffxz"] Oct 30 17:03:02.969971 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:02.969927 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81b60d6e-bef0-499e-9cb5-7c935fc0587e" path="/var/lib/kubelet/pods/81b60d6e-bef0-499e-9cb5-7c935fc0587e/volumes" Oct 30 17:03:08.621821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.618715 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws"] Oct 30 17:03:08.621821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.619662 6400 cpu_manager.go:414] "containerMap: removing container" podUID="28e0f55b-622b-4ecf-bb99-5b080c872f4c" containerName="main" Oct 30 17:03:08.621821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.619689 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28e0f55b-622b-4ecf-bb99-5b080c872f4c" containerName="main" Oct 30 17:03:08.621821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.619749 6400 cpu_manager.go:414] "containerMap: removing container" podUID="81b60d6e-bef0-499e-9cb5-7c935fc0587e" containerName="main" Oct 30 17:03:08.621821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.619756 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81b60d6e-bef0-499e-9cb5-7c935fc0587e" containerName="main" Oct 30 17:03:08.621821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.619933 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="28e0f55b-622b-4ecf-bb99-5b080c872f4c" containerName="main" Oct 30 17:03:08.621821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.619955 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="81b60d6e-bef0-499e-9cb5-7c935fc0587e" containerName="main" Oct 30 17:03:08.639421 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.638262 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws"] Oct 30 17:03:08.639421 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.638382 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.656956 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.656929 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-ae5afd-serving-cert\"" Oct 30 17:03:08.806627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.805629 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvtxn\" (UniqueName: \"kubernetes.io/projected/00a9feb0-d258-4ab0-9d89-82f30f869e6f-kube-api-access-fvtxn\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.806627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.805726 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.806627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.805891 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.806627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.806040 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a9feb0-d258-4ab0-9d89-82f30f869e6f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.907158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.907074 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvtxn\" (UniqueName: \"kubernetes.io/projected/00a9feb0-d258-4ab0-9d89-82f30f869e6f-kube-api-access-fvtxn\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.907158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.907145 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.907373 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.907260 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.907428 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.907395 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a9feb0-d258-4ab0-9d89-82f30f869e6f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.908055 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.908027 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a9feb0-d258-4ab0-9d89-82f30f869e6f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.909979 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.909952 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.909979 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.909972 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.928543 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.928512 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvtxn\" (UniqueName: \"kubernetes.io/projected/00a9feb0-d258-4ab0-9d89-82f30f869e6f-kube-api-access-fvtxn\") pod \"maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:08.979196 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:08.979172 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:09.284549 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:09.284519 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws"] Oct 30 17:03:09.286110 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:03:09.286081 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00a9feb0_d258_4ab0_9d89_82f30f869e6f.slice/crio-92f6c853d745c68fc8396df0e04cea258f2b27d7e17e8cf271c28ad4730cafcb WatchSource:0}: Error finding container 92f6c853d745c68fc8396df0e04cea258f2b27d7e17e8cf271c28ad4730cafcb: Status 404 returned error can't find the container with id 92f6c853d745c68fc8396df0e04cea258f2b27d7e17e8cf271c28ad4730cafcb Oct 30 17:03:09.287453 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:09.287437 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:03:09.782450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:09.781947 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" event={"ID":"00a9feb0-d258-4ab0-9d89-82f30f869e6f","Type":"ContainerStarted","Data":"92f6c853d745c68fc8396df0e04cea258f2b27d7e17e8cf271c28ad4730cafcb"} Oct 30 17:03:09.782450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:09.782193 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:09.824877 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:09.824823 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" podStartSLOduration=1.419168786 podStartE2EDuration="1.824809203s" totalImagesPullingTime="405.640417ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:09.821342234 +0000 UTC m=+2498.971623223" watchObservedRunningTime="2026-10-30 17:03:09.824809203 +0000 UTC m=+2498.975090190" Oct 30 17:03:10.791185 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:10.791148 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" event={"ID":"00a9feb0-d258-4ab0-9d89-82f30f869e6f","Type":"ContainerStarted","Data":"6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2"} Oct 30 17:03:16.803996 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:16.803965 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:03:17.469177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.469143 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9"] Oct 30 17:03:17.473585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.473556 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.493907 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.493882 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9"] Oct 30 17:03:17.498776 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.498753 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-dc74d5\"/\"rate-test-model-dc74d5-kserve-self-signed-certs\"" Oct 30 17:03:17.499043 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.499026 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-dc74d5\"/\"kube-root-ca.crt\"" Oct 30 17:03:17.499641 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.499584 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-dc74d5\"/\"default-dockercfg-lthgr\"" Oct 30 17:03:17.499745 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.499732 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-dc74d5\"/\"openshift-service-ca.crt\"" Oct 30 17:03:17.538614 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.538577 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-tmp-dir\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.538737 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.538644 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-model-cache\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.538845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.538826 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjqnm\" (UniqueName: \"kubernetes.io/projected/09583221-74c8-4466-a342-2910378ff059-kube-api-access-mjqnm\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.538918 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.538904 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09583221-74c8-4466-a342-2910378ff059-tls-certs\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.539003 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.538992 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-home\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.539066 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.539051 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-dshm\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.639949 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.639917 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-dshm\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.640144 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.639976 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-tmp-dir\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.640144 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.640088 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-model-cache\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.640293 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.640271 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjqnm\" (UniqueName: \"kubernetes.io/projected/09583221-74c8-4466-a342-2910378ff059-kube-api-access-mjqnm\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.640389 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.640375 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09583221-74c8-4466-a342-2910378ff059-tls-certs\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.640450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.640278 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-tmp-dir\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.640522 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.640486 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-home\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.640855 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.640828 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-home\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.640977 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.640880 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-model-cache\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.642574 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.642547 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-dshm\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.642957 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.642941 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09583221-74c8-4466-a342-2910378ff059-tls-certs\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.661510 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.661488 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjqnm\" (UniqueName: \"kubernetes.io/projected/09583221-74c8-4466-a342-2910378ff059-kube-api-access-mjqnm\") pod \"rate-test-model-dc74d5-kserve-5759bc457c-864j9\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:17.796177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:17.796138 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:18.124122 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:18.124090 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9"] Oct 30 17:03:18.124556 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:03:18.124522 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09583221_74c8_4466_a342_2910378ff059.slice/crio-c19d9457a2e0918cab2c13f3b400ec5095b8d3b19f98d2610116ced9b8b6ee6d WatchSource:0}: Error finding container c19d9457a2e0918cab2c13f3b400ec5095b8d3b19f98d2610116ced9b8b6ee6d: Status 404 returned error can't find the container with id c19d9457a2e0918cab2c13f3b400ec5095b8d3b19f98d2610116ced9b8b6ee6d Oct 30 17:03:18.854043 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:18.854000 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" event={"ID":"09583221-74c8-4466-a342-2910378ff059","Type":"ContainerStarted","Data":"5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee"} Oct 30 17:03:18.854221 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:18.854055 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" event={"ID":"09583221-74c8-4466-a342-2910378ff059","Type":"ContainerStarted","Data":"c19d9457a2e0918cab2c13f3b400ec5095b8d3b19f98d2610116ced9b8b6ee6d"} Oct 30 17:03:18.890636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:18.890570 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" podStartSLOduration=1.890553804 podStartE2EDuration="1.890553804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:03:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:03:18.890118137 +0000 UTC m=+2508.040399124" watchObservedRunningTime="2026-10-30 17:03:18.890553804 +0000 UTC m=+2508.040834795" Oct 30 17:03:19.796686 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:19.796659 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:24.599764 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:03:24.599726 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:03:27.797026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:27.796995 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:27.809754 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:27.809730 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:27.928456 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:03:27.928349 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:03:54.602276 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:03:54.602232 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:04:21.956532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:21.956495 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8956-25e9f6"] Oct 30 17:04:21.960884 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:21.960848 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:21.976655 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:21.976624 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8956-25e9f6"] Oct 30 17:04:21.982562 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:21.982536 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-xr4c7\"" Oct 30 17:04:22.015996 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:22.015956 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj7d7\" (UniqueName: \"kubernetes.io/projected/23eadb62-8e0b-4e17-9191-407fd835db49-kube-api-access-hj7d7\") pod \"test-curl-8956-25e9f6\" (UID: \"23eadb62-8e0b-4e17-9191-407fd835db49\") " pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:22.117615 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:22.117558 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj7d7\" (UniqueName: \"kubernetes.io/projected/23eadb62-8e0b-4e17-9191-407fd835db49-kube-api-access-hj7d7\") pod \"test-curl-8956-25e9f6\" (UID: \"23eadb62-8e0b-4e17-9191-407fd835db49\") " pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:22.139044 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:22.139012 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj7d7\" (UniqueName: \"kubernetes.io/projected/23eadb62-8e0b-4e17-9191-407fd835db49-kube-api-access-hj7d7\") pod \"test-curl-8956-25e9f6\" (UID: \"23eadb62-8e0b-4e17-9191-407fd835db49\") " pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:22.274706 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:22.274638 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:22.584700 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:22.584619 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8956-25e9f6"] Oct 30 17:04:22.591774 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:04:22.591747 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23eadb62_8e0b_4e17_9191_407fd835db49.slice/crio-67327aabae7ef658a67117a646a7d683950823467c9666810499c3c82405aa1d WatchSource:0}: Error finding container 67327aabae7ef658a67117a646a7d683950823467c9666810499c3c82405aa1d: Status 404 returned error can't find the container with id 67327aabae7ef658a67117a646a7d683950823467c9666810499c3c82405aa1d Oct 30 17:04:23.407061 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:23.407026 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8956-25e9f6" event={"ID":"23eadb62-8e0b-4e17-9191-407fd835db49","Type":"ContainerStarted","Data":"f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15"} Oct 30 17:04:23.407061 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:23.407075 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8956-25e9f6" event={"ID":"23eadb62-8e0b-4e17-9191-407fd835db49","Type":"ContainerStarted","Data":"67327aabae7ef658a67117a646a7d683950823467c9666810499c3c82405aa1d"} Oct 30 17:04:23.441706 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:23.441649 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8956-25e9f6" podStartSLOduration=2.441632916 podStartE2EDuration="2.441632916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:23.439760672 +0000 UTC m=+2572.590041661" watchObservedRunningTime="2026-10-30 17:04:23.441632916 +0000 UTC m=+2572.591913903" Oct 30 17:04:24.607731 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:24.607687 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:04:25.177655 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.177615 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n"] Oct 30 17:04:25.182879 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.182854 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.196069 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.196039 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b4214a\"/\"kube-root-ca.crt\"" Oct 30 17:04:25.198499 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.198475 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w1-b4214a\"/\"openshift-service-ca.crt\"" Oct 30 17:04:25.210649 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.208338 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b4214a\"/\"default-dockercfg-4nftw\"" Oct 30 17:04:25.210649 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.208391 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w1-b4214a\"/\"auth-test-model-b4214a-kserve-self-signed-certs\"" Oct 30 17:04:25.213004 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.212977 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n"] Oct 30 17:04:25.263252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.263211 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-dshm\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.263452 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.263266 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-model-cache\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.263452 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.263411 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rts44\" (UniqueName: \"kubernetes.io/projected/8431187c-9a0c-4ea5-854a-b4496f5cd277-kube-api-access-rts44\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.263588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.263571 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-home\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.263652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.263596 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8431187c-9a0c-4ea5-854a-b4496f5cd277-tls-certs\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.263775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.263749 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-tmp-dir\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.365067 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.365023 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-home\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.365067 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.365060 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8431187c-9a0c-4ea5-854a-b4496f5cd277-tls-certs\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.365331 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.365125 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-tmp-dir\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.365331 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.365178 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-dshm\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.365331 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.365203 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-model-cache\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.365331 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.365283 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rts44\" (UniqueName: \"kubernetes.io/projected/8431187c-9a0c-4ea5-854a-b4496f5cd277-kube-api-access-rts44\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.365554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.365499 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-home\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.365638 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.365549 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-tmp-dir\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.365832 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.365810 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-model-cache\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.367791 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.367770 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-dshm\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.367962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.367943 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8431187c-9a0c-4ea5-854a-b4496f5cd277-tls-certs\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.386473 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.386438 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rts44\" (UniqueName: \"kubernetes.io/projected/8431187c-9a0c-4ea5-854a-b4496f5cd277-kube-api-access-rts44\") pod \"auth-test-model-b4214a-kserve-5499bc7957-5zd6n\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.508637 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.504918 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:25.822352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:25.822327 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n"] Oct 30 17:04:25.824724 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:04:25.824692 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8431187c_9a0c_4ea5_854a_b4496f5cd277.slice/crio-181cd0bb48c6dc011656f72f87c01a669e5bb3c3b7c7da4e5e76e415f33c67c0 WatchSource:0}: Error finding container 181cd0bb48c6dc011656f72f87c01a669e5bb3c3b7c7da4e5e76e415f33c67c0: Status 404 returned error can't find the container with id 181cd0bb48c6dc011656f72f87c01a669e5bb3c3b7c7da4e5e76e415f33c67c0 Oct 30 17:04:26.434168 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:26.434124 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" event={"ID":"8431187c-9a0c-4ea5-854a-b4496f5cd277","Type":"ContainerStarted","Data":"8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43"} Oct 30 17:04:26.434353 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:26.434179 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" event={"ID":"8431187c-9a0c-4ea5-854a-b4496f5cd277","Type":"ContainerStarted","Data":"181cd0bb48c6dc011656f72f87c01a669e5bb3c3b7c7da4e5e76e415f33c67c0"} Oct 30 17:04:26.473118 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:26.473067 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" podStartSLOduration=1.473053463 podStartE2EDuration="1.473053463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:26.469720771 +0000 UTC m=+2575.620001759" watchObservedRunningTime="2026-10-30 17:04:26.473053463 +0000 UTC m=+2575.623334451" Oct 30 17:04:26.505821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:26.505792 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:30.219901 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.219235 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8956-25e9f6"] Oct 30 17:04:30.219901 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.219459 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8956-25e9f6" podUID="23eadb62-8e0b-4e17-9191-407fd835db49" containerName="test-curl-8956-25e9f6" containerID="cri-o://f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15" gracePeriod=2 Oct 30 17:04:30.224175 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.224144 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8956-25e9f6"] Oct 30 17:04:30.232708 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:30.232670 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8956-25e9f6\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="23eadb62-8e0b-4e17-9191-407fd835db49" pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:30.380070 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.380032 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-8956-01d266"] Oct 30 17:04:30.380642 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.380626 6400 cpu_manager.go:414] "containerMap: removing container" podUID="23eadb62-8e0b-4e17-9191-407fd835db49" containerName="test-curl-8956-25e9f6" Oct 30 17:04:30.380689 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.380643 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23eadb62-8e0b-4e17-9191-407fd835db49" containerName="test-curl-8956-25e9f6" Oct 30 17:04:30.380795 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.380786 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="23eadb62-8e0b-4e17-9191-407fd835db49" containerName="test-curl-8956-25e9f6" Oct 30 17:04:30.384205 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.384189 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8956-01d266" Oct 30 17:04:30.399954 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.399928 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8956-01d266"] Oct 30 17:04:30.431886 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:30.431841 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8956-25e9f6\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="23eadb62-8e0b-4e17-9191-407fd835db49" pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:30.453737 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.453705 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glwrp\" (UniqueName: \"kubernetes.io/projected/f023e80e-4bd9-4b44-94ed-b1a1b012084b-kube-api-access-glwrp\") pod \"test-curl-8956-01d266\" (UID: \"f023e80e-4bd9-4b44-94ed-b1a1b012084b\") " pod="opendatahub/test-curl-8956-01d266" Oct 30 17:04:30.555098 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.555064 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glwrp\" (UniqueName: \"kubernetes.io/projected/f023e80e-4bd9-4b44-94ed-b1a1b012084b-kube-api-access-glwrp\") pod \"test-curl-8956-01d266\" (UID: \"f023e80e-4bd9-4b44-94ed-b1a1b012084b\") " pod="opendatahub/test-curl-8956-01d266" Oct 30 17:04:30.579101 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.579072 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-glwrp\" (UniqueName: \"kubernetes.io/projected/f023e80e-4bd9-4b44-94ed-b1a1b012084b-kube-api-access-glwrp\") pod \"test-curl-8956-01d266\" (UID: \"f023e80e-4bd9-4b44-94ed-b1a1b012084b\") " pod="opendatahub/test-curl-8956-01d266" Oct 30 17:04:30.699080 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:30.699043 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8956-01d266" Oct 30 17:04:30.979654 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:30.979539 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8956-25e9f6\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="23eadb62-8e0b-4e17-9191-407fd835db49" pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:31.007726 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:31.007701 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-8956-01d266"] Oct 30 17:04:31.008967 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:04:31.008943 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf023e80e_4bd9_4b44_94ed_b1a1b012084b.slice/crio-f902d07922e17f34ad723e667c72819bf61711032a8e5edd4a9cefb506946893 WatchSource:0}: Error finding container f902d07922e17f34ad723e667c72819bf61711032a8e5edd4a9cefb506946893: Status 404 returned error can't find the container with id f902d07922e17f34ad723e667c72819bf61711032a8e5edd4a9cefb506946893 Oct 30 17:04:31.472164 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:31.472123 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8956-01d266" event={"ID":"f023e80e-4bd9-4b44-94ed-b1a1b012084b","Type":"ContainerStarted","Data":"7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672"} Oct 30 17:04:31.472689 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:31.472175 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-8956-01d266" event={"ID":"f023e80e-4bd9-4b44-94ed-b1a1b012084b","Type":"ContainerStarted","Data":"f902d07922e17f34ad723e667c72819bf61711032a8e5edd4a9cefb506946893"} Oct 30 17:04:31.511067 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:31.511009 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-8956-01d266" podStartSLOduration=1.510995298 podStartE2EDuration="1.510995298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:31.508366462 +0000 UTC m=+2580.658647449" watchObservedRunningTime="2026-10-30 17:04:31.510995298 +0000 UTC m=+2580.661276286" Oct 30 17:04:32.277335 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:32.277299 6400 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23eadb62_8e0b_4e17_9191_407fd835db49.slice/crio-f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23eadb62_8e0b_4e17_9191_407fd835db49.slice/crio-67327aabae7ef658a67117a646a7d683950823467c9666810499c3c82405aa1d\": RecentStats: unable to find data in memory cache]" Oct 30 17:04:32.277335 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:32.277313 6400 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23eadb62_8e0b_4e17_9191_407fd835db49.slice/crio-f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23eadb62_8e0b_4e17_9191_407fd835db49.slice/crio-67327aabae7ef658a67117a646a7d683950823467c9666810499c3c82405aa1d\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23eadb62_8e0b_4e17_9191_407fd835db49.slice/crio-conmon-f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:04:32.277543 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:32.277464 6400 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23eadb62_8e0b_4e17_9191_407fd835db49.slice/crio-f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23eadb62_8e0b_4e17_9191_407fd835db49.slice/crio-conmon-f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23eadb62_8e0b_4e17_9191_407fd835db49.slice/crio-67327aabae7ef658a67117a646a7d683950823467c9666810499c3c82405aa1d\": RecentStats: unable to find data in memory cache]" Oct 30 17:04:32.395838 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.395804 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-8956-01d266"] Oct 30 17:04:32.400953 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.400921 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-8956-01d266"] Oct 30 17:04:32.467999 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.467967 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:32.481771 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.481681 6400 generic.go:356] "Generic (PLEG): container finished" podID="23eadb62-8e0b-4e17-9191-407fd835db49" containerID="f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15" exitCode=137 Oct 30 17:04:32.481771 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.481739 6400 scope.go:122] "RemoveContainer" containerID="f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15" Oct 30 17:04:32.481771 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.481750 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:32.492886 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.492852 6400 scope.go:122] "RemoveContainer" containerID="f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15" Oct 30 17:04:32.493198 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:32.493176 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15\": container with ID starting with f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15 not found: ID does not exist" containerID="f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15" Oct 30 17:04:32.493296 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.493207 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15"} err="failed to get container status \"f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15\": rpc error: code = NotFound desc = could not find container \"f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15\": container with ID starting with f1a6afcbc5e9481e2aef704f0ce1fd1cb172df3b4adc84b0ab32f8d10b111b15 not found: ID does not exist" Oct 30 17:04:32.586070 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.585291 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23eadb62-8e0b-4e17-9191-407fd835db49-kube-api-access-hj7d7\" (UniqueName: \"kubernetes.io/projected/23eadb62-8e0b-4e17-9191-407fd835db49-kube-api-access-hj7d7\") pod \"23eadb62-8e0b-4e17-9191-407fd835db49\" (UID: \"23eadb62-8e0b-4e17-9191-407fd835db49\") " Oct 30 17:04:32.588648 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.588590 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23eadb62-8e0b-4e17-9191-407fd835db49-kube-api-access-hj7d7" pod "23eadb62-8e0b-4e17-9191-407fd835db49" (UID: "23eadb62-8e0b-4e17-9191-407fd835db49"). InnerVolumeSpecName "kube-api-access-hj7d7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:32.689035 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.687448 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj7d7\" (UniqueName: \"kubernetes.io/projected/23eadb62-8e0b-4e17-9191-407fd835db49-kube-api-access-hj7d7\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:04:32.807384 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:32.807337 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8956-25e9f6\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="23eadb62-8e0b-4e17-9191-407fd835db49" pod="opendatahub/test-curl-8956-25e9f6" Oct 30 17:04:32.972712 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:32.972677 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23eadb62-8e0b-4e17-9191-407fd835db49" path="/var/lib/kubelet/pods/23eadb62-8e0b-4e17-9191-407fd835db49/volumes" Oct 30 17:04:33.491473 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:33.491436 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-8956-01d266" podUID="f023e80e-4bd9-4b44-94ed-b1a1b012084b" containerName="test-curl-8956-01d266" containerID="cri-o://7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672" gracePeriod=2 Oct 30 17:04:35.505563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:35.505535 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:35.519758 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:35.519733 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:35.722153 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:35.722127 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8956-01d266" Oct 30 17:04:35.837329 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:35.837236 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f023e80e-4bd9-4b44-94ed-b1a1b012084b-kube-api-access-glwrp\" (UniqueName: \"kubernetes.io/projected/f023e80e-4bd9-4b44-94ed-b1a1b012084b-kube-api-access-glwrp\") pod \"f023e80e-4bd9-4b44-94ed-b1a1b012084b\" (UID: \"f023e80e-4bd9-4b44-94ed-b1a1b012084b\") " Oct 30 17:04:35.839705 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:35.839677 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f023e80e-4bd9-4b44-94ed-b1a1b012084b-kube-api-access-glwrp" pod "f023e80e-4bd9-4b44-94ed-b1a1b012084b" (UID: "f023e80e-4bd9-4b44-94ed-b1a1b012084b"). InnerVolumeSpecName "kube-api-access-glwrp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:04:35.939548 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:35.939518 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glwrp\" (UniqueName: \"kubernetes.io/projected/f023e80e-4bd9-4b44-94ed-b1a1b012084b-kube-api-access-glwrp\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:04:36.517719 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:36.517688 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:04:36.523007 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:36.522976 6400 generic.go:356] "Generic (PLEG): container finished" podID="f023e80e-4bd9-4b44-94ed-b1a1b012084b" containerID="7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672" exitCode=137 Oct 30 17:04:36.523153 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:36.523025 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-8956-01d266" Oct 30 17:04:36.523153 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:36.523071 6400 scope.go:122] "RemoveContainer" containerID="7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672" Oct 30 17:04:36.536027 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:36.536006 6400 scope.go:122] "RemoveContainer" containerID="7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672" Oct 30 17:04:36.536372 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:36.536346 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672\": container with ID starting with 7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672 not found: ID does not exist" containerID="7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672" Oct 30 17:04:36.536469 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:36.536378 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672"} err="failed to get container status \"7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672\": rpc error: code = NotFound desc = could not find container \"7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672\": container with ID starting with 7e2b1f0799d2a317aaf28f9d9ee240302d9d7653fd7d91b62a5cdcb2521b1672 not found: ID does not exist" Oct 30 17:04:36.582549 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:36.582507 6400 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-8956-01d266\" is forbidden: User \"system:node:ip-10-0-128-181.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-128-181.ec2.internal' and this object" podUID="f023e80e-4bd9-4b44-94ed-b1a1b012084b" pod="opendatahub/test-curl-8956-01d266" Oct 30 17:04:36.968173 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:36.968097 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f023e80e-4bd9-4b44-94ed-b1a1b012084b" path="/var/lib/kubelet/pods/f023e80e-4bd9-4b44-94ed-b1a1b012084b/volumes" Oct 30 17:04:38.142668 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.142597 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g"] Oct 30 17:04:38.143468 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.143447 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f023e80e-4bd9-4b44-94ed-b1a1b012084b" containerName="test-curl-8956-01d266" Oct 30 17:04:38.143588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.143470 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f023e80e-4bd9-4b44-94ed-b1a1b012084b" containerName="test-curl-8956-01d266" Oct 30 17:04:38.143708 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.143687 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f023e80e-4bd9-4b44-94ed-b1a1b012084b" containerName="test-curl-8956-01d266" Oct 30 17:04:38.148656 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.148633 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.162783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.162764 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-05810b-openshift-default-dockercfg-cf5d9\"" Oct 30 17:04:38.178374 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.178343 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g"] Oct 30 17:04:38.275480 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.275440 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.275742 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.275486 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8b49\" (UniqueName: \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-kube-api-access-g8b49\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.275742 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.275662 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.275742 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.275692 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.275877 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.275859 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.275937 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.275914 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.276012 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.275999 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.276052 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.276025 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.276104 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.276070 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05edb360-ea8e-4b72-846d-e3ecd317bb37-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377088 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377056 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377357 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377127 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377357 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377149 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377357 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377199 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05edb360-ea8e-4b72-846d-e3ecd317bb37-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377357 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377307 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377357 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377338 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8b49\" (UniqueName: \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-kube-api-access-g8b49\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377407 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377435 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377541 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377571 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-certs\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377619 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-data\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.377725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377658 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-credential-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.378006 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.377931 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-socket\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.378049 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.378035 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05edb360-ea8e-4b72-846d-e3ecd317bb37-istiod-ca-cert\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.379886 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.379859 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-podinfo\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.380003 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.379985 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-envoy\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.400746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.400678 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-token\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.400868 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.400824 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8b49\" (UniqueName: \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-kube-api-access-g8b49\") pod \"e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.463076 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.463033 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:38.788515 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.788476 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g"] Oct 30 17:04:38.789706 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:04:38.789676 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05edb360_ea8e_4b72_846d_e3ecd317bb37.slice/crio-f31c463d319c02ae99ca152accd4a402fd07871fdbebd7e06f829890e94a0da6 WatchSource:0}: Error finding container f31c463d319c02ae99ca152accd4a402fd07871fdbebd7e06f829890e94a0da6: Status 404 returned error can't find the container with id f31c463d319c02ae99ca152accd4a402fd07871fdbebd7e06f829890e94a0da6 Oct 30 17:04:38.799521 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.799475 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:38.799663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.799568 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:38.799718 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:38.799659 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:39.554693 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:39.554658 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" event={"ID":"05edb360-ea8e-4b72-846d-e3ecd317bb37","Type":"ContainerStarted","Data":"26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f"} Oct 30 17:04:39.555090 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:39.554720 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" event={"ID":"05edb360-ea8e-4b72-846d-e3ecd317bb37","Type":"ContainerStarted","Data":"f31c463d319c02ae99ca152accd4a402fd07871fdbebd7e06f829890e94a0da6"} Oct 30 17:04:39.626968 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:39.626911 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" podStartSLOduration=1.62689171 podStartE2EDuration="1.62689171s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:39.613675165 +0000 UTC m=+2588.763956147" watchObservedRunningTime="2026-10-30 17:04:39.62689171 +0000 UTC m=+2588.777172703" Oct 30 17:04:40.463146 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:40.463114 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:40.465281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:40.465253 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:04:40.465427 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:40.465301 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:04:41.464201 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.464166 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:04:41.464668 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.464232 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:04:41.838833 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.838788 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn"] Oct 30 17:04:41.843302 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.843279 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:41.850147 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.850117 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn"] Oct 30 17:04:41.862879 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.862855 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-9e533f-openshift-default-dockercfg-x62c6\"" Oct 30 17:04:41.949853 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.949817 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:41.950046 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.949986 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:41.950046 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.950038 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:41.950141 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.950125 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:41.950215 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.950202 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:41.950262 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.950238 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:41.950395 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.950369 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctzvm\" (UniqueName: \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-kube-api-access-ctzvm\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:41.950466 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.950410 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d6bcdd-9469-4086-af49-e19ff94217e9-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:41.950517 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:41.950481 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052242 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052455 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052336 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052501 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052459 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052538 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052511 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052573 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052559 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052657 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052642 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052714 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052689 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052772 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052721 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-certs\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052772 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052739 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-data\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052906 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052807 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctzvm\" (UniqueName: \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-kube-api-access-ctzvm\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.052906 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.052834 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d6bcdd-9469-4086-af49-e19ff94217e9-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.053035 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.053014 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-credential-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.053282 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.053257 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-socket\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.053522 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.053505 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d6bcdd-9469-4086-af49-e19ff94217e9-istiod-ca-cert\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.055251 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.055226 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-podinfo\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.055346 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.055248 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-envoy\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.075760 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.075734 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-token\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.075904 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.075826 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctzvm\" (UniqueName: \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-kube-api-access-ctzvm\") pod \"e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.167593 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.167502 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:42.464211 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.464136 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:04:42.464211 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.464189 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:04:42.540013 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.539980 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn"] Oct 30 17:04:42.541507 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:04:42.541467 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d6bcdd_9469_4086_af49_e19ff94217e9.slice/crio-401b5b4e627b7fbe4fdb266571f197cce223f603cb5ec88abe5dc0d3c5657195 WatchSource:0}: Error finding container 401b5b4e627b7fbe4fdb266571f197cce223f603cb5ec88abe5dc0d3c5657195: Status 404 returned error can't find the container with id 401b5b4e627b7fbe4fdb266571f197cce223f603cb5ec88abe5dc0d3c5657195 Oct 30 17:04:42.545416 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.545380 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:42.545497 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.545466 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:42.545550 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.545495 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:04:42.583072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:42.583041 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" event={"ID":"61d6bcdd-9469-4086-af49-e19ff94217e9","Type":"ContainerStarted","Data":"401b5b4e627b7fbe4fdb266571f197cce223f603cb5ec88abe5dc0d3c5657195"} Oct 30 17:04:43.464632 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:43.463786 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:04:43.464632 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:43.463858 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:04:43.595735 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:43.595702 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" event={"ID":"61d6bcdd-9469-4086-af49-e19ff94217e9","Type":"ContainerStarted","Data":"8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db"} Oct 30 17:04:43.684185 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:43.683654 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" podStartSLOduration=2.683641007 podStartE2EDuration="2.683641007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:04:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:04:43.680929858 +0000 UTC m=+2592.831210849" watchObservedRunningTime="2026-10-30 17:04:43.683641007 +0000 UTC m=+2592.833921995" Oct 30 17:04:44.168724 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:44.168683 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:44.171380 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:44.170272 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:04:44.171380 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:44.170314 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 17:04:44.464511 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:44.464001 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:04:44.464511 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:44.464065 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:04:45.167944 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:45.167902 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:04:45.168223 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:45.167966 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 17:04:45.464028 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:45.463942 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" start-of-body= Oct 30 17:04:45.464028 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:45.464012 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.157:15021/healthz/ready\": dial tcp 10.134.0.157:15021: connect: connection refused" Oct 30 17:04:45.598961 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:45.598927 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx"] Oct 30 17:04:46.167877 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:46.167840 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:04:46.168061 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:46.167898 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 17:04:46.469030 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:46.468949 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:46.469349 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:46.469325 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:46.469862 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:46.469844 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:04:47.168083 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:47.168050 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:04:47.168481 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:47.168103 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 17:04:48.168768 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:48.168732 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:04:48.169274 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:48.168796 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 17:04:49.168653 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:49.168615 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" start-of-body= Oct 30 17:04:49.168828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:49.168699 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.158:15021/healthz/ready\": dial tcp 10.134.0.158:15021: connect: connection refused" Oct 30 17:04:50.173226 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:50.173191 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:50.173663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:50.173643 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:50.174414 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:04:50.174396 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:04:54.606576 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:04:54.606529 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:05:00.796031 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:00.795963 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" podUID="33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" containerName="main" containerID="cri-o://2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267" gracePeriod=45 Oct 30 17:05:00.969128 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:00.969092 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" podUID="33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.148:8000/ready\": dial tcp 10.134.0.148:8000: connect: connection refused" Oct 30 17:05:01.098581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.098556 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:05:01.177202 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.177158 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-home\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-home\") pod \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " Oct 30 17:05:01.177413 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.177316 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-model-cache\") pod \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " Oct 30 17:05:01.177496 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.177453 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-home" pod "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" (UID: "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:01.177496 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.177479 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tmp-dir\") pod \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " Oct 30 17:05:01.177672 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.177528 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-dshm\") pod \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " Oct 30 17:05:01.177672 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.177582 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-model-cache" pod "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" (UID: "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:01.177828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.177633 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-kube-api-access-q5wl7\" (UniqueName: \"kubernetes.io/projected/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-kube-api-access-q5wl7\") pod \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " Oct 30 17:05:01.177828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.177694 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tmp-dir" pod "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" (UID: "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:01.177946 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.177832 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tls-certs\" (UniqueName: \"kubernetes.io/secret/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tls-certs\") pod \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\" (UID: \"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48\") " Oct 30 17:05:01.178421 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.178398 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:01.178421 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.178420 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:01.178680 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.178429 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:01.180657 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.180615 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-kube-api-access-q5wl7" pod "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" (UID: "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48"). InnerVolumeSpecName "kube-api-access-q5wl7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:01.180833 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.180692 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-dshm" pod "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" (UID: "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:01.181060 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.181034 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tls-certs" pod "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" (UID: "33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:01.280108 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.280075 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:01.280108 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.280105 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5wl7\" (UniqueName: \"kubernetes.io/projected/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-kube-api-access-q5wl7\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:01.280108 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.280116 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:01.785135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.785102 6400 generic.go:356] "Generic (PLEG): container finished" podID="33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" containerID="2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267" exitCode=0 Oct 30 17:05:01.785332 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.785158 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" Oct 30 17:05:01.785332 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.785180 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" event={"ID":"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48","Type":"ContainerDied","Data":"2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267"} Oct 30 17:05:01.785332 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.785224 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx" event={"ID":"33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48","Type":"ContainerDied","Data":"e1e0d891a6603e359909800d95ae2cfb9a0589f079aacb840add163f659238d4"} Oct 30 17:05:01.785332 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.785244 6400 scope.go:122] "RemoveContainer" containerID="2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267" Oct 30 17:05:01.796931 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.796796 6400 scope.go:122] "RemoveContainer" containerID="2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267" Oct 30 17:05:01.797158 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:05:01.797082 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267\": container with ID starting with 2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267 not found: ID does not exist" containerID="2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267" Oct 30 17:05:01.797158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.797111 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267"} err="failed to get container status \"2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267\": rpc error: code = NotFound desc = could not find container \"2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267\": container with ID starting with 2cd8eb6ad124905bb06b791db1629cf5211bd0d5d9abda6fbc3249ac5efde267 not found: ID does not exist" Oct 30 17:05:01.842694 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.842659 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx"] Oct 30 17:05:01.852254 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:01.852225 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-bfecdd/auth-test-model-bfecdd-kserve-7b5c887fc7-kzgxx"] Oct 30 17:05:02.969306 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:02.969275 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" path="/var/lib/kubelet/pods/33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48/volumes" Oct 30 17:05:11.930199 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:11.930056 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk"] Oct 30 17:05:11.930931 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:11.930909 6400 cpu_manager.go:414] "containerMap: removing container" podUID="33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" containerName="main" Oct 30 17:05:11.931044 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:11.930934 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" containerName="main" Oct 30 17:05:11.931161 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:11.931148 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="33e1f0a0-69e7-4cfc-b76f-ca88e55c3d48" containerName="main" Oct 30 17:05:11.936214 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:11.936195 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:11.954755 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:11.954724 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk"] Oct 30 17:05:11.962310 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:11.962279 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-b16279-serving-cert\"" Oct 30 17:05:12.036425 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.036390 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-metrics-certs\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.036610 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.036451 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.036610 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.036561 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a348964-7aa9-4ca7-98b2-62992f766d4b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.036696 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.036625 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dfl8\" (UniqueName: \"kubernetes.io/projected/7a348964-7aa9-4ca7-98b2-62992f766d4b-kube-api-access-9dfl8\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.162807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.162747 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.163054 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.162859 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a348964-7aa9-4ca7-98b2-62992f766d4b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.163054 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.162909 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dfl8\" (UniqueName: \"kubernetes.io/projected/7a348964-7aa9-4ca7-98b2-62992f766d4b-kube-api-access-9dfl8\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.163299 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.163267 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-metrics-certs\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.165923 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.165731 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a348964-7aa9-4ca7-98b2-62992f766d4b-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.170023 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.169996 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-metrics-certs\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.171534 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.171506 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-maas-api-tls\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.188520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.188456 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dfl8\" (UniqueName: \"kubernetes.io/projected/7a348964-7aa9-4ca7-98b2-62992f766d4b-kube-api-access-9dfl8\") pod \"maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.253674 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.253640 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:12.562997 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.562969 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk"] Oct 30 17:05:12.563740 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:05:12.563709 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a348964_7aa9_4ca7_98b2_62992f766d4b.slice/crio-c1c144b2235e20ce10506bee371cad94b201063f5bd399d63ab0c58adcabe140 WatchSource:0}: Error finding container c1c144b2235e20ce10506bee371cad94b201063f5bd399d63ab0c58adcabe140: Status 404 returned error can't find the container with id c1c144b2235e20ce10506bee371cad94b201063f5bd399d63ab0c58adcabe140 Oct 30 17:05:12.887689 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:12.887550 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" event={"ID":"7a348964-7aa9-4ca7-98b2-62992f766d4b","Type":"ContainerStarted","Data":"c1c144b2235e20ce10506bee371cad94b201063f5bd399d63ab0c58adcabe140"} Oct 30 17:05:13.896562 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:13.896530 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" event={"ID":"7a348964-7aa9-4ca7-98b2-62992f766d4b","Type":"ContainerStarted","Data":"855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70"} Oct 30 17:05:13.897010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:13.896615 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:13.938718 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:13.938658 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" podStartSLOduration=2.47666351 podStartE2EDuration="2.938639351s" totalImagesPullingTime="461.975841ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:13.936921336 +0000 UTC m=+2623.087202325" watchObservedRunningTime="2026-10-30 17:05:13.938639351 +0000 UTC m=+2623.088920339" Oct 30 17:05:16.118660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.118622 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54"] Oct 30 17:05:16.124743 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.124708 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.150143 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.150108 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-ae5afd\"/\"kube-root-ca.crt\"" Oct 30 17:05:16.150627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.150144 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w1-ae5afd\"/\"openshift-service-ca.crt\"" Oct 30 17:05:16.150627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.150226 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-ae5afd\"/\"default-dockercfg-nj7b7\"" Oct 30 17:05:16.150627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.150239 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w1-ae5afd\"/\"auth-test-model-ae5afd-kserve-self-signed-certs\"" Oct 30 17:05:16.154250 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.154216 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54"] Oct 30 17:05:16.221246 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.221206 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-model-cache\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.221535 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.221322 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-dshm\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.221535 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.221422 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tmp-dir\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.221707 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.221552 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-home\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.221707 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.221656 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tls-certs\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.221821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.221716 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85rxs\" (UniqueName: \"kubernetes.io/projected/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-kube-api-access-85rxs\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.323340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.323302 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-model-cache\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.323542 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.323347 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-dshm\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.323542 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.323495 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tmp-dir\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.323697 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.323656 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-home\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.323758 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.323717 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tls-certs\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.323815 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.323766 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85rxs\" (UniqueName: \"kubernetes.io/projected/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-kube-api-access-85rxs\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.323909 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.323886 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-model-cache\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.323971 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.323907 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tmp-dir\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.324016 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.323963 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-home\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.325852 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.325828 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-dshm\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.326491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.326473 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tls-certs\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.347932 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.347900 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-85rxs\" (UniqueName: \"kubernetes.io/projected/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-kube-api-access-85rxs\") pod \"auth-test-model-ae5afd-kserve-76dff69645-w8b54\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.442641 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.442519 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:16.780163 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.774179 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54"] Oct 30 17:05:16.922548 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.922513 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" event={"ID":"88d766ab-40c9-4c3f-be73-b3d4f4d933b5","Type":"ContainerStarted","Data":"162c9ac6afbeba031a7b12bfef141daede487cbec0f0d9bd8f46ce71f48aa132"} Oct 30 17:05:16.922790 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.922558 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" event={"ID":"88d766ab-40c9-4c3f-be73-b3d4f4d933b5","Type":"ContainerStarted","Data":"8fb33c345144b2613c9d2ff0994779b2848aa54a842a837ec39ff9d6ef1379e7"} Oct 30 17:05:16.964989 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:16.964926 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" podStartSLOduration=0.964908434 podStartE2EDuration="964.908434ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:05:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:05:16.962429001 +0000 UTC m=+2626.112709991" watchObservedRunningTime="2026-10-30 17:05:16.964908434 +0000 UTC m=+2626.115189423" Oct 30 17:05:17.443175 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:17.443132 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:19.910845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:19.910814 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:05:24.641502 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:05:24.641459 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:05:26.443798 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:26.443767 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:26.462176 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:26.462153 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:27.008842 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:27.008809 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:05:42.014454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:42.014421 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r"] Oct 30 17:05:42.425274 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:42.423259 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7"] Oct 30 17:05:54.613219 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:05:54.613178 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:05:57.088105 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.088058 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" podUID="1ddbbe4a-21cf-4281-94ce-490b0469e0ce" containerName="main" containerID="cri-o://31f95bf97210552264db07ad249d13e4606c5aa5354f96cd52fb81475bf38d36" gracePeriod=45 Oct 30 17:05:57.322736 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.322703 6400 generic.go:356] "Generic (PLEG): container finished" podID="1ddbbe4a-21cf-4281-94ce-490b0469e0ce" containerID="31f95bf97210552264db07ad249d13e4606c5aa5354f96cd52fb81475bf38d36" exitCode=0 Oct 30 17:05:57.322918 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.322776 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" event={"ID":"1ddbbe4a-21cf-4281-94ce-490b0469e0ce","Type":"ContainerDied","Data":"31f95bf97210552264db07ad249d13e4606c5aa5354f96cd52fb81475bf38d36"} Oct 30 17:05:57.369268 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.369243 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:05:57.420789 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.420759 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tls-certs\") pod \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " Oct 30 17:05:57.420978 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.420837 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-home\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-home\") pod \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " Oct 30 17:05:57.420978 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.420890 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-model-cache\") pod \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " Oct 30 17:05:57.421531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.420986 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tmp-dir\") pod \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " Oct 30 17:05:57.421531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.421182 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-dshm\") pod \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " Oct 30 17:05:57.421531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.421264 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-kube-api-access-hqvxv\" (UniqueName: \"kubernetes.io/projected/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-kube-api-access-hqvxv\") pod \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\" (UID: \"1ddbbe4a-21cf-4281-94ce-490b0469e0ce\") " Oct 30 17:05:57.421531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.421413 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-model-cache" pod "1ddbbe4a-21cf-4281-94ce-490b0469e0ce" (UID: "1ddbbe4a-21cf-4281-94ce-490b0469e0ce"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:57.421531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.421449 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-home" pod "1ddbbe4a-21cf-4281-94ce-490b0469e0ce" (UID: "1ddbbe4a-21cf-4281-94ce-490b0469e0ce"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:57.421944 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.421639 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tmp-dir" pod "1ddbbe4a-21cf-4281-94ce-490b0469e0ce" (UID: "1ddbbe4a-21cf-4281-94ce-490b0469e0ce"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:57.422087 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.422025 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:57.422087 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.422051 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:57.422087 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.422066 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:57.424993 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.424953 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-dshm" pod "1ddbbe4a-21cf-4281-94ce-490b0469e0ce" (UID: "1ddbbe4a-21cf-4281-94ce-490b0469e0ce"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:57.424993 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.424973 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tls-certs" pod "1ddbbe4a-21cf-4281-94ce-490b0469e0ce" (UID: "1ddbbe4a-21cf-4281-94ce-490b0469e0ce"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:57.425172 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.425035 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-kube-api-access-hqvxv" pod "1ddbbe4a-21cf-4281-94ce-490b0469e0ce" (UID: "1ddbbe4a-21cf-4281-94ce-490b0469e0ce"). InnerVolumeSpecName "kube-api-access-hqvxv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:57.523255 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.523220 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:57.523255 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.523248 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:57.523255 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.523258 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hqvxv\" (UniqueName: \"kubernetes.io/projected/1ddbbe4a-21cf-4281-94ce-490b0469e0ce-kube-api-access-hqvxv\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:57.665177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.665065 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" podUID="0fd9b015-c869-45c1-9482-fc0e46acf69d" containerName="main" containerID="cri-o://567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d" gracePeriod=45 Oct 30 17:05:57.930816 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:57.930790 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:05:58.029948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.029913 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-model-cache\") pod \"0fd9b015-c869-45c1-9482-fc0e46acf69d\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " Oct 30 17:05:58.030148 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.030076 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0fd9b015-c869-45c1-9482-fc0e46acf69d-tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd9b015-c869-45c1-9482-fc0e46acf69d-tls-certs\") pod \"0fd9b015-c869-45c1-9482-fc0e46acf69d\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " Oct 30 17:05:58.030148 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.030130 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0fd9b015-c869-45c1-9482-fc0e46acf69d-kube-api-access-pxd7h\" (UniqueName: \"kubernetes.io/projected/0fd9b015-c869-45c1-9482-fc0e46acf69d-kube-api-access-pxd7h\") pod \"0fd9b015-c869-45c1-9482-fc0e46acf69d\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " Oct 30 17:05:58.030295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.030199 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-home\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-home\") pod \"0fd9b015-c869-45c1-9482-fc0e46acf69d\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " Oct 30 17:05:58.030295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.030231 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-tmp-dir\") pod \"0fd9b015-c869-45c1-9482-fc0e46acf69d\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " Oct 30 17:05:58.030295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.030232 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-model-cache" pod "0fd9b015-c869-45c1-9482-fc0e46acf69d" (UID: "0fd9b015-c869-45c1-9482-fc0e46acf69d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:58.030469 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.030305 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-dshm\") pod \"0fd9b015-c869-45c1-9482-fc0e46acf69d\" (UID: \"0fd9b015-c869-45c1-9482-fc0e46acf69d\") " Oct 30 17:05:58.030469 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.030453 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-tmp-dir" pod "0fd9b015-c869-45c1-9482-fc0e46acf69d" (UID: "0fd9b015-c869-45c1-9482-fc0e46acf69d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:58.030585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.030487 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-home" pod "0fd9b015-c869-45c1-9482-fc0e46acf69d" (UID: "0fd9b015-c869-45c1-9482-fc0e46acf69d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:58.031398 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.031211 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:58.031398 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.031238 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:58.031398 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.031252 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:58.033022 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.032994 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd9b015-c869-45c1-9482-fc0e46acf69d-kube-api-access-pxd7h" pod "0fd9b015-c869-45c1-9482-fc0e46acf69d" (UID: "0fd9b015-c869-45c1-9482-fc0e46acf69d"). InnerVolumeSpecName "kube-api-access-pxd7h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:05:58.033139 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.033110 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-dshm" pod "0fd9b015-c869-45c1-9482-fc0e46acf69d" (UID: "0fd9b015-c869-45c1-9482-fc0e46acf69d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:05:58.033212 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.033198 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0fd9b015-c869-45c1-9482-fc0e46acf69d-tls-certs" pod "0fd9b015-c869-45c1-9482-fc0e46acf69d" (UID: "0fd9b015-c869-45c1-9482-fc0e46acf69d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:05:58.132700 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.132669 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0fd9b015-c869-45c1-9482-fc0e46acf69d-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:58.132700 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.132695 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pxd7h\" (UniqueName: \"kubernetes.io/projected/0fd9b015-c869-45c1-9482-fc0e46acf69d-kube-api-access-pxd7h\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:58.132700 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.132705 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0fd9b015-c869-45c1-9482-fc0e46acf69d-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:05:58.331542 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.331507 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" Oct 30 17:05:58.331542 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.331535 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r" event={"ID":"1ddbbe4a-21cf-4281-94ce-490b0469e0ce","Type":"ContainerDied","Data":"bb162acb016968bed08bdcd6e87987bb052105e131d301d065e9550c692b4641"} Oct 30 17:05:58.331827 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.331586 6400 scope.go:122] "RemoveContainer" containerID="31f95bf97210552264db07ad249d13e4606c5aa5354f96cd52fb81475bf38d36" Oct 30 17:05:58.333390 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.333361 6400 generic.go:356] "Generic (PLEG): container finished" podID="0fd9b015-c869-45c1-9482-fc0e46acf69d" containerID="567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d" exitCode=0 Oct 30 17:05:58.333522 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.333404 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" event={"ID":"0fd9b015-c869-45c1-9482-fc0e46acf69d","Type":"ContainerDied","Data":"567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d"} Oct 30 17:05:58.333522 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.333431 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" event={"ID":"0fd9b015-c869-45c1-9482-fc0e46acf69d","Type":"ContainerDied","Data":"9ea0660afe91ce904fd51240cb47586b776604473e0c07b2afcbdfc53ec8c977"} Oct 30 17:05:58.333522 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.333406 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" Oct 30 17:05:58.344294 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.344273 6400 scope.go:122] "RemoveContainer" containerID="567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d" Oct 30 17:05:58.355008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.354991 6400 scope.go:122] "RemoveContainer" containerID="567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d" Oct 30 17:05:58.355297 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:05:58.355279 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d\": container with ID starting with 567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d not found: ID does not exist" containerID="567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d" Oct 30 17:05:58.355360 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.355307 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d"} err="failed to get container status \"567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d\": rpc error: code = NotFound desc = could not find container \"567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d\": container with ID starting with 567c3851823a85d8abe9734a79f9f5db076e2c10a5c293f9af6cd118c1d5f86d not found: ID does not exist" Oct 30 17:05:58.384250 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.384220 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r"] Oct 30 17:05:58.395437 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.395409 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-05810b/sub-test-model-05810b-kserve-57dc5f5b85-4ww8r"] Oct 30 17:05:58.420902 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.420861 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7"] Oct 30 17:05:58.424713 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.424689 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7"] Oct 30 17:05:58.919868 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.919827 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w4-3ed10a/rate-test-model-3ed10a-kserve-7bb48d77-8fvh7" podUID="0fd9b015-c869-45c1-9482-fc0e46acf69d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.146:8000/ready\": context deadline exceeded" Oct 30 17:05:58.968393 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.968358 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0fd9b015-c869-45c1-9482-fc0e46acf69d" path="/var/lib/kubelet/pods/0fd9b015-c869-45c1-9482-fc0e46acf69d/volumes" Oct 30 17:05:58.968877 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:05:58.968863 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ddbbe4a-21cf-4281-94ce-490b0469e0ce" path="/var/lib/kubelet/pods/1ddbbe4a-21cf-4281-94ce-490b0469e0ce/volumes" Oct 30 17:06:13.001623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:13.001574 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn"] Oct 30 17:06:24.602802 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:06:24.602760 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:06:28.030345 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.030305 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" podUID="93da5f58-2c4f-4753-b177-26a7f86d8eb4" containerName="main" containerID="cri-o://22bb7915341b5d298cb18f298c2b5789e2e1c650ea584515044e162d07b3ca2d" gracePeriod=45 Oct 30 17:06:28.187011 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.186111 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" podUID="93da5f58-2c4f-4753-b177-26a7f86d8eb4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.150:8000/ready\": dial tcp 10.134.0.150:8000: connect: connection refused" Oct 30 17:06:28.567206 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.567166 6400 generic.go:356] "Generic (PLEG): container finished" podID="93da5f58-2c4f-4753-b177-26a7f86d8eb4" containerID="22bb7915341b5d298cb18f298c2b5789e2e1c650ea584515044e162d07b3ca2d" exitCode=0 Oct 30 17:06:28.567448 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.567258 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" event={"ID":"93da5f58-2c4f-4753-b177-26a7f86d8eb4","Type":"ContainerDied","Data":"22bb7915341b5d298cb18f298c2b5789e2e1c650ea584515044e162d07b3ca2d"} Oct 30 17:06:28.567448 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.567311 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" event={"ID":"93da5f58-2c4f-4753-b177-26a7f86d8eb4","Type":"ContainerDied","Data":"b3fa6a5fb3a6d12f9723ad6c4e8daa7c491f9c458e75cc2919949ef2fb40a2f6"} Oct 30 17:06:28.567448 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.567329 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3fa6a5fb3a6d12f9723ad6c4e8daa7c491f9c458e75cc2919949ef2fb40a2f6" Oct 30 17:06:28.604039 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.604013 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:06:28.676640 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.671106 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tmp-dir\") pod \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " Oct 30 17:06:28.676640 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.671203 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93da5f58-2c4f-4753-b177-26a7f86d8eb4-kube-api-access-47bmx\" (UniqueName: \"kubernetes.io/projected/93da5f58-2c4f-4753-b177-26a7f86d8eb4-kube-api-access-47bmx\") pod \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " Oct 30 17:06:28.676640 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.671311 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-model-cache\") pod \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " Oct 30 17:06:28.676640 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.671348 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-home\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-home\") pod \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " Oct 30 17:06:28.676640 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.671467 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tls-certs\" (UniqueName: \"kubernetes.io/secret/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tls-certs\") pod \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " Oct 30 17:06:28.676640 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.671558 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-dshm\") pod \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\" (UID: \"93da5f58-2c4f-4753-b177-26a7f86d8eb4\") " Oct 30 17:06:28.676640 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.672816 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-model-cache" pod "93da5f58-2c4f-4753-b177-26a7f86d8eb4" (UID: "93da5f58-2c4f-4753-b177-26a7f86d8eb4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:28.676640 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.673045 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tmp-dir" pod "93da5f58-2c4f-4753-b177-26a7f86d8eb4" (UID: "93da5f58-2c4f-4753-b177-26a7f86d8eb4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:28.676640 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.675863 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-home" pod "93da5f58-2c4f-4753-b177-26a7f86d8eb4" (UID: "93da5f58-2c4f-4753-b177-26a7f86d8eb4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:28.687044 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.687012 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93da5f58-2c4f-4753-b177-26a7f86d8eb4-kube-api-access-47bmx" pod "93da5f58-2c4f-4753-b177-26a7f86d8eb4" (UID: "93da5f58-2c4f-4753-b177-26a7f86d8eb4"). InnerVolumeSpecName "kube-api-access-47bmx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:06:28.715691 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.694995 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-dshm" pod "93da5f58-2c4f-4753-b177-26a7f86d8eb4" (UID: "93da5f58-2c4f-4753-b177-26a7f86d8eb4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:06:28.715691 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.695494 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tls-certs" pod "93da5f58-2c4f-4753-b177-26a7f86d8eb4" (UID: "93da5f58-2c4f-4753-b177-26a7f86d8eb4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:06:28.774819 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.774783 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:06:28.774819 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.774817 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:06:28.774819 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.774830 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:06:28.775158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.774841 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:06:28.775158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.774853 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93da5f58-2c4f-4753-b177-26a7f86d8eb4-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:06:28.775158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:28.774869 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47bmx\" (UniqueName: \"kubernetes.io/projected/93da5f58-2c4f-4753-b177-26a7f86d8eb4-kube-api-access-47bmx\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:06:29.575343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.575253 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn" Oct 30 17:06:29.628228 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.628196 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn"] Oct 30 17:06:29.639661 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.639628 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w4-7c4b84/rate-test-model-7c4b84-kserve-744c4d69b5-8rhbn"] Oct 30 17:06:29.658536 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.658506 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb"] Oct 30 17:06:29.659150 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.659135 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1ddbbe4a-21cf-4281-94ce-490b0469e0ce" containerName="main" Oct 30 17:06:29.659215 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.659151 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ddbbe4a-21cf-4281-94ce-490b0469e0ce" containerName="main" Oct 30 17:06:29.659215 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.659190 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0fd9b015-c869-45c1-9482-fc0e46acf69d" containerName="main" Oct 30 17:06:29.659215 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.659196 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0fd9b015-c869-45c1-9482-fc0e46acf69d" containerName="main" Oct 30 17:06:29.659215 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.659211 6400 cpu_manager.go:414] "containerMap: removing container" podUID="93da5f58-2c4f-4753-b177-26a7f86d8eb4" containerName="main" Oct 30 17:06:29.659215 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.659215 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93da5f58-2c4f-4753-b177-26a7f86d8eb4" containerName="main" Oct 30 17:06:29.659475 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.659298 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0fd9b015-c869-45c1-9482-fc0e46acf69d" containerName="main" Oct 30 17:06:29.659475 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.659309 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="93da5f58-2c4f-4753-b177-26a7f86d8eb4" containerName="main" Oct 30 17:06:29.659475 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.659317 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ddbbe4a-21cf-4281-94ce-490b0469e0ce" containerName="main" Oct 30 17:06:29.668032 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.668012 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.685139 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.685115 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-55e83e-serving-cert\"" Oct 30 17:06:29.687399 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.687374 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb"] Oct 30 17:06:29.796207 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.796175 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-metrics-certs\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.796407 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.796324 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dgww\" (UniqueName: \"kubernetes.io/projected/d700bf64-2ff5-4546-92e1-c433fc8d3a61-kube-api-access-9dgww\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.796407 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.796373 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d700bf64-2ff5-4546-92e1-c433fc8d3a61-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.796532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.796405 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-maas-api-tls\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.898583 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.898490 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dgww\" (UniqueName: \"kubernetes.io/projected/d700bf64-2ff5-4546-92e1-c433fc8d3a61-kube-api-access-9dgww\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.898583 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.898554 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d700bf64-2ff5-4546-92e1-c433fc8d3a61-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.898894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.898589 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-maas-api-tls\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.898894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.898775 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-metrics-certs\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.899354 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.899330 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d700bf64-2ff5-4546-92e1-c433fc8d3a61-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.901457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.901432 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-maas-api-tls\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.901544 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.901465 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-metrics-certs\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.922726 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.922687 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dgww\" (UniqueName: \"kubernetes.io/projected/d700bf64-2ff5-4546-92e1-c433fc8d3a61-kube-api-access-9dgww\") pod \"maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:29.982574 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:29.982538 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:30.311707 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:30.310588 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb"] Oct 30 17:06:30.584277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:30.584183 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" event={"ID":"d700bf64-2ff5-4546-92e1-c433fc8d3a61","Type":"ContainerStarted","Data":"6e80434cb7ee0bc3a63938e3d5ad00120409e744ec8904336bea5238a2c4f3ea"} Oct 30 17:06:30.972133 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:30.972102 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93da5f58-2c4f-4753-b177-26a7f86d8eb4" path="/var/lib/kubelet/pods/93da5f58-2c4f-4753-b177-26a7f86d8eb4/volumes" Oct 30 17:06:31.598347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:31.598304 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" event={"ID":"d700bf64-2ff5-4546-92e1-c433fc8d3a61","Type":"ContainerStarted","Data":"277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c"} Oct 30 17:06:31.598921 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:31.598400 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:31.639716 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:31.639655 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" podStartSLOduration=2.028515951 podStartE2EDuration="2.639636968s" totalImagesPullingTime="611.121017ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:31.638692772 +0000 UTC m=+2700.788973765" watchObservedRunningTime="2026-10-30 17:06:31.639636968 +0000 UTC m=+2700.789917957" Oct 30 17:06:32.030365 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:32.030326 6400 scope.go:122] "RemoveContainer" containerID="c0b350e60902475d660a2c52527b977f6d28290ff2b6d0c2931195115cece5c0" Oct 30 17:06:37.612450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:37.612422 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:06:38.825686 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:38.818206 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24"] Oct 30 17:06:38.836631 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:38.829296 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:06:38.846628 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:38.845187 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-b16279-dockercfg-mp5nj\"" Oct 30 17:06:38.847481 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:38.847451 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24"] Oct 30 17:06:38.849777 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:38.849752 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-b16279\"" Oct 30 17:06:38.938916 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:38.938884 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bt27c\" (UniqueName: \"kubernetes.io/projected/aa869356-749f-4c10-b350-f1aa6fc87baf-kube-api-access-bt27c\") pod \"payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24\" (UID: \"aa869356-749f-4c10-b350-f1aa6fc87baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:06:38.939120 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:38.939059 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa869356-749f-4c10-b350-f1aa6fc87baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24\" (UID: \"aa869356-749f-4c10-b350-f1aa6fc87baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:06:38.956361 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:38.956336 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw"] Oct 30 17:06:38.960824 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:38.960798 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:06:38.984457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:38.984429 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw"] Oct 30 17:06:39.040537 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.040481 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bt27c\" (UniqueName: \"kubernetes.io/projected/aa869356-749f-4c10-b350-f1aa6fc87baf-kube-api-access-bt27c\") pod \"payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24\" (UID: \"aa869356-749f-4c10-b350-f1aa6fc87baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:06:39.040785 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.040586 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2scxr\" (UniqueName: \"kubernetes.io/projected/f12e857f-ab74-4122-abe5-5c0297070fc0-kube-api-access-2scxr\") pod \"payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw\" (UID: \"f12e857f-ab74-4122-abe5-5c0297070fc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:06:39.040883 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.040780 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa869356-749f-4c10-b350-f1aa6fc87baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24\" (UID: \"aa869356-749f-4c10-b350-f1aa6fc87baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:06:39.040998 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.040979 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12e857f-ab74-4122-abe5-5c0297070fc0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw\" (UID: \"f12e857f-ab74-4122-abe5-5c0297070fc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:06:39.041701 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.041678 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa869356-749f-4c10-b350-f1aa6fc87baf-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24\" (UID: \"aa869356-749f-4c10-b350-f1aa6fc87baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:06:39.063479 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.063443 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bt27c\" (UniqueName: \"kubernetes.io/projected/aa869356-749f-4c10-b350-f1aa6fc87baf-kube-api-access-bt27c\") pod \"payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24\" (UID: \"aa869356-749f-4c10-b350-f1aa6fc87baf\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:06:39.142343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.142263 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12e857f-ab74-4122-abe5-5c0297070fc0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw\" (UID: \"f12e857f-ab74-4122-abe5-5c0297070fc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:06:39.142490 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.142370 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2scxr\" (UniqueName: \"kubernetes.io/projected/f12e857f-ab74-4122-abe5-5c0297070fc0-kube-api-access-2scxr\") pod \"payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw\" (UID: \"f12e857f-ab74-4122-abe5-5c0297070fc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:06:39.142987 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.142967 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12e857f-ab74-4122-abe5-5c0297070fc0-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw\" (UID: \"f12e857f-ab74-4122-abe5-5c0297070fc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:06:39.165349 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.165319 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2scxr\" (UniqueName: \"kubernetes.io/projected/f12e857f-ab74-4122-abe5-5c0297070fc0-kube-api-access-2scxr\") pod \"payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw\" (UID: \"f12e857f-ab74-4122-abe5-5c0297070fc0\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:06:39.169116 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.169084 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:06:39.275694 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.275660 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:06:39.478999 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.478972 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24"] Oct 30 17:06:39.479829 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:06:39.479803 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa869356_749f_4c10_b350_f1aa6fc87baf.slice/crio-6e40c8e4fc97f3f890becae73fb525fe6ec366e8154c192d7ce532ac3e6ff756 WatchSource:0}: Error finding container 6e40c8e4fc97f3f890becae73fb525fe6ec366e8154c192d7ce532ac3e6ff756: Status 404 returned error can't find the container with id 6e40c8e4fc97f3f890becae73fb525fe6ec366e8154c192d7ce532ac3e6ff756 Oct 30 17:06:39.605794 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.605763 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw"] Oct 30 17:06:39.606954 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:06:39.606928 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf12e857f_ab74_4122_abe5_5c0297070fc0.slice/crio-9773671e55bfc59632caff8075c981bee9ded77ee7acf83d4c08d1e82fe248b1 WatchSource:0}: Error finding container 9773671e55bfc59632caff8075c981bee9ded77ee7acf83d4c08d1e82fe248b1: Status 404 returned error can't find the container with id 9773671e55bfc59632caff8075c981bee9ded77ee7acf83d4c08d1e82fe248b1 Oct 30 17:06:39.665266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.665215 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" event={"ID":"aa869356-749f-4c10-b350-f1aa6fc87baf","Type":"ContainerStarted","Data":"6e40c8e4fc97f3f890becae73fb525fe6ec366e8154c192d7ce532ac3e6ff756"} Oct 30 17:06:39.666956 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:39.666926 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" event={"ID":"f12e857f-ab74-4122-abe5-5c0297070fc0","Type":"ContainerStarted","Data":"9773671e55bfc59632caff8075c981bee9ded77ee7acf83d4c08d1e82fe248b1"} Oct 30 17:06:41.688587 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:41.688546 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" event={"ID":"f12e857f-ab74-4122-abe5-5c0297070fc0","Type":"ContainerStarted","Data":"b8f442d3198ce32f9433ede0b554afee649f63a2deb5b5490ffc2dc6a4243da7"} Oct 30 17:06:41.689169 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:41.688671 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:06:41.690473 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:41.690451 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" event={"ID":"aa869356-749f-4c10-b350-f1aa6fc87baf","Type":"ContainerStarted","Data":"33ac69cf10e7eaa4f822a7c6809c174b289923321b1ed0aca954bfd89d402f39"} Oct 30 17:06:41.690639 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:41.690590 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:06:41.729762 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:41.729713 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" podStartSLOduration=2.410400188 podStartE2EDuration="3.729697991s" totalImagesPullingTime="1.319297803s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:41.728453497 +0000 UTC m=+2710.878734489" watchObservedRunningTime="2026-10-30 17:06:41.729697991 +0000 UTC m=+2710.879978978" Oct 30 17:06:41.769074 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:41.769007 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" podStartSLOduration=2.277758526 podStartE2EDuration="3.768987901s" totalImagesPullingTime="1.491229375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:41.767828267 +0000 UTC m=+2710.918109256" watchObservedRunningTime="2026-10-30 17:06:41.768987901 +0000 UTC m=+2710.919268890" Oct 30 17:06:41.977528 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:41.977446 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m"] Oct 30 17:06:41.983014 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:41.982991 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.000343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.000312 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9e533f\"/\"kube-root-ca.crt\"" Oct 30 17:06:42.000545 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.000359 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9e533f\"/\"sub-test-model-9e533f-kserve-self-signed-certs\"" Oct 30 17:06:42.000545 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.000420 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-9e533f\"/\"openshift-service-ca.crt\"" Oct 30 17:06:42.000545 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.000424 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-9e533f\"/\"default-dockercfg-8j2k2\"" Oct 30 17:06:42.001959 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.001936 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m"] Oct 30 17:06:42.093766 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.093730 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-dshm\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.093944 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.093863 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-model-cache\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.093989 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.093964 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-home\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.093989 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.093983 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14738f12-f84d-4bf4-9c03-916c866acb95-tls-certs\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.094159 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.094137 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vk2vq\" (UniqueName: \"kubernetes.io/projected/14738f12-f84d-4bf4-9c03-916c866acb95-kube-api-access-vk2vq\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.094232 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.094195 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-tmp-dir\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.195652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.195581 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-home\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.195652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.195660 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14738f12-f84d-4bf4-9c03-916c866acb95-tls-certs\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.195925 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.195773 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vk2vq\" (UniqueName: \"kubernetes.io/projected/14738f12-f84d-4bf4-9c03-916c866acb95-kube-api-access-vk2vq\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.195925 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.195848 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-tmp-dir\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.196045 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.195974 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-dshm\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.196104 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.196067 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-model-cache\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.196104 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.196077 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-home\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.196220 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.196154 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-tmp-dir\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.196411 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.196384 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-model-cache\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.198452 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.198428 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-dshm\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.198583 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.198530 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14738f12-f84d-4bf4-9c03-916c866acb95-tls-certs\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.220418 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.220382 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vk2vq\" (UniqueName: \"kubernetes.io/projected/14738f12-f84d-4bf4-9c03-916c866acb95-kube-api-access-vk2vq\") pod \"sub-test-model-9e533f-kserve-587cffd96c-w446m\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.300628 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.299876 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:42.639373 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.639283 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m"] Oct 30 17:06:42.647852 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:06:42.646880 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod14738f12_f84d_4bf4_9c03_916c866acb95.slice/crio-68f8d666d0efa4ab0c853504d6b5c79b61583fb3afe29478a7c4f51ea5e32e19 WatchSource:0}: Error finding container 68f8d666d0efa4ab0c853504d6b5c79b61583fb3afe29478a7c4f51ea5e32e19: Status 404 returned error can't find the container with id 68f8d666d0efa4ab0c853504d6b5c79b61583fb3afe29478a7c4f51ea5e32e19 Oct 30 17:06:42.699674 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:42.699637 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" event={"ID":"14738f12-f84d-4bf4-9c03-916c866acb95","Type":"ContainerStarted","Data":"68f8d666d0efa4ab0c853504d6b5c79b61583fb3afe29478a7c4f51ea5e32e19"} Oct 30 17:06:43.709100 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:43.709051 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" event={"ID":"14738f12-f84d-4bf4-9c03-916c866acb95","Type":"ContainerStarted","Data":"80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1"} Oct 30 17:06:43.753818 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:43.753755 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" podStartSLOduration=2.753736525 podStartE2EDuration="2.753736525s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:06:43.748556963 +0000 UTC m=+2712.898837952" watchObservedRunningTime="2026-10-30 17:06:43.753736525 +0000 UTC m=+2712.904017516" Oct 30 17:06:44.300354 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:44.300315 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:52.301134 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:52.301093 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:52.313906 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:52.313880 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:52.702407 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:52.702330 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:06:52.702407 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:52.702402 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:06:52.787842 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:52.787816 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:06:54.603611 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:06:54.603559 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:06:59.313038 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.313001 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr"] Oct 30 17:06:59.318455 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.318427 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.340102 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.340072 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr"] Oct 30 17:06:59.346707 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.346672 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-7c4b84-openshift-default-dockercfg-5sg2r\"" Oct 30 17:06:59.364402 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.364369 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-envoy\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.364574 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.364486 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.364688 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.364588 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.364753 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.364718 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-data\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.364812 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.364768 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-certs\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.364873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.364809 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-credential-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.364873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.364841 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-podinfo\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.364873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.364865 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsqsn\" (UniqueName: \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-kube-api-access-zsqsn\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.365040 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.364953 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-token\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.466396 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.466363 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-credential-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.466396 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.466395 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-podinfo\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.466739 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.466425 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsqsn\" (UniqueName: \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-kube-api-access-zsqsn\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.466739 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.466488 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-token\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.466739 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.466552 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-envoy\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.466739 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.466648 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.466973 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.466780 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.466973 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.466810 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-data\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.466973 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.466849 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-certs\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.467139 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.466814 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-credential-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.467139 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.467104 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-socket\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.467293 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.467271 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-data\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.467517 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.467434 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-certs\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.467680 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.467662 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istiod-ca-cert\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.469141 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.469104 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-envoy\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.469744 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.469716 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-podinfo\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.495287 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.495252 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-token\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.495512 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.495487 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsqsn\" (UniqueName: \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-kube-api-access-zsqsn\") pod \"e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.637280 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.637202 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:06:59.985125 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.985086 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:06:59.985257 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.985181 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:06:59.985257 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:06:59.985228 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:07:00.005064 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:00.005023 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr"] Oct 30 17:07:00.862249 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:00.862206 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" event={"ID":"3aae2a95-9ae0-456d-8115-8084d9dc7ab6","Type":"ContainerStarted","Data":"6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347"} Oct 30 17:07:00.862786 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:00.862287 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" event={"ID":"3aae2a95-9ae0-456d-8115-8084d9dc7ab6","Type":"ContainerStarted","Data":"60b64b89555d3f119db9486068a71d830635b22c38335750717c407e46f75bdf"} Oct 30 17:07:00.940774 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:00.940027 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" podStartSLOduration=1.9400053879999999 podStartE2EDuration="1.940005388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:06:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:00.918162035 +0000 UTC m=+2730.068443024" watchObservedRunningTime="2026-10-30 17:07:00.940005388 +0000 UTC m=+2730.090286378" Oct 30 17:07:01.637631 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:01.637580 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:07:01.639329 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:01.639301 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 17:07:01.639463 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:01.639364 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" podUID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 17:07:02.639629 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:02.638994 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 17:07:02.639629 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:02.639070 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" podUID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 17:07:03.637996 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:03.637951 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 17:07:03.638213 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:03.638017 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" podUID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 17:07:04.648822 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:04.648785 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 17:07:04.650717 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:04.650683 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" podUID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 17:07:05.639425 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:05.639276 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 30 17:07:05.639425 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:05.639355 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" podUID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 30 17:07:06.643100 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:06.643065 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:07:06.643623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:06.643494 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:07:06.644222 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:06.644203 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:07:12.356661 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.356595 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt"] Oct 30 17:07:12.361071 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.361052 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.391982 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.388299 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-330282-serving-cert\"" Oct 30 17:07:12.395230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.395193 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt"] Oct 30 17:07:12.462781 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.462741 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.462999 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.462982 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-metrics-certs\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.463046 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.463014 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkcs2\" (UniqueName: \"kubernetes.io/projected/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-kube-api-access-kkcs2\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.463046 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.463038 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.564527 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.564492 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.564770 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.564670 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-metrics-certs\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.564770 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.564698 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkcs2\" (UniqueName: \"kubernetes.io/projected/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-kube-api-access-kkcs2\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.564770 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.564722 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.565300 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.565274 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.567404 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.567384 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-metrics-certs\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.567472 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.567420 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-maas-api-tls\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.588128 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.588096 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkcs2\" (UniqueName: \"kubernetes.io/projected/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-kube-api-access-kkcs2\") pod \"maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.675710 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.675578 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:12.984177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:12.984149 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt"] Oct 30 17:07:12.985243 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:07:12.985217 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4d0a60d_f6d5_4322_89cf_4c226de2b2e0.slice/crio-e77a6daedf1c93f2bfa8639fa8634ae41d6acafc9d5b33db39849a8927e2045a WatchSource:0}: Error finding container e77a6daedf1c93f2bfa8639fa8634ae41d6acafc9d5b33db39849a8927e2045a: Status 404 returned error can't find the container with id e77a6daedf1c93f2bfa8639fa8634ae41d6acafc9d5b33db39849a8927e2045a Oct 30 17:07:13.962547 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:13.962510 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" event={"ID":"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0","Type":"ContainerStarted","Data":"32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018"} Oct 30 17:07:13.962985 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:13.962589 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" event={"ID":"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0","Type":"ContainerStarted","Data":"e77a6daedf1c93f2bfa8639fa8634ae41d6acafc9d5b33db39849a8927e2045a"} Oct 30 17:07:13.962985 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:13.962660 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:14.018477 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:14.018424 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" podStartSLOduration=1.5757650490000001 podStartE2EDuration="2.018410232s" totalImagesPullingTime="442.645183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:14.00327678 +0000 UTC m=+2743.153557770" watchObservedRunningTime="2026-10-30 17:07:14.018410232 +0000 UTC m=+2743.168691239" Oct 30 17:07:19.977414 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:19.977386 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:07:24.602053 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:07:24.602016 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:07:24.905914 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:24.905825 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm"] Oct 30 17:07:24.906112 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:24.906065 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" containerID="cri-o://98afc09d83e03534b36ab82bb94d70b5fd35667a5ef2d3817d7c57b9c089f611" gracePeriod=30 Oct 30 17:07:27.222954 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:27.222919 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9"] Oct 30 17:07:28.787247 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:28.787214 6400 patch_prober.go:29] interesting pod/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:07:28.787673 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:28.787276 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:07:30.100185 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.100149 6400 generic.go:356] "Generic (PLEG): container finished" podID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerID="98afc09d83e03534b36ab82bb94d70b5fd35667a5ef2d3817d7c57b9c089f611" exitCode=0 Oct 30 17:07:30.100679 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.100319 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" event={"ID":"649f1900-0d41-45c8-9cee-e1e6c7fd2610","Type":"ContainerDied","Data":"98afc09d83e03534b36ab82bb94d70b5fd35667a5ef2d3817d7c57b9c089f611"} Oct 30 17:07:30.187818 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.187736 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 17:07:30.281882 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.281830 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-certs\") pod \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " Oct 30 17:07:30.282095 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282055 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-credential-socket\") pod \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " Oct 30 17:07:30.282095 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282091 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-podinfo\") pod \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " Oct 30 17:07:30.282230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282139 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-certs" pod "649f1900-0d41-45c8-9cee-e1e6c7fd2610" (UID: "649f1900-0d41-45c8-9cee-e1e6c7fd2610"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:30.282230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282126 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-data\") pod \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " Oct 30 17:07:30.282337 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282254 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-kube-api-access-bkrmj\" (UniqueName: \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-kube-api-access-bkrmj\") pod \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " Oct 30 17:07:30.282397 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282331 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-token\" (UniqueName: \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-token\") pod \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " Oct 30 17:07:30.282397 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282338 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-credential-socket" pod "649f1900-0d41-45c8-9cee-e1e6c7fd2610" (UID: "649f1900-0d41-45c8-9cee-e1e6c7fd2610"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:30.282525 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282417 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-data" pod "649f1900-0d41-45c8-9cee-e1e6c7fd2610" (UID: "649f1900-0d41-45c8-9cee-e1e6c7fd2610"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:30.282670 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282642 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-socket\") pod \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " Oct 30 17:07:30.282762 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282700 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istiod-ca-cert\") pod \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " Oct 30 17:07:30.282841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282769 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-envoy\") pod \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\" (UID: \"649f1900-0d41-45c8-9cee-e1e6c7fd2610\") " Oct 30 17:07:30.282995 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.282958 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-socket" pod "649f1900-0d41-45c8-9cee-e1e6c7fd2610" (UID: "649f1900-0d41-45c8-9cee-e1e6c7fd2610"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:30.283970 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.283508 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istiod-ca-cert" pod "649f1900-0d41-45c8-9cee-e1e6c7fd2610" (UID: "649f1900-0d41-45c8-9cee-e1e6c7fd2610"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:07:30.283970 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.283879 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:30.283970 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.283904 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:30.283970 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.283919 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:30.283970 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.283934 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:30.283970 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.283946 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:30.285619 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.285577 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-podinfo" pod "649f1900-0d41-45c8-9cee-e1e6c7fd2610" (UID: "649f1900-0d41-45c8-9cee-e1e6c7fd2610"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:07:30.285721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.285660 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-kube-api-access-bkrmj" pod "649f1900-0d41-45c8-9cee-e1e6c7fd2610" (UID: "649f1900-0d41-45c8-9cee-e1e6c7fd2610"). InnerVolumeSpecName "kube-api-access-bkrmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:30.286021 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.285999 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-token" pod "649f1900-0d41-45c8-9cee-e1e6c7fd2610" (UID: "649f1900-0d41-45c8-9cee-e1e6c7fd2610"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:30.287026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.287005 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-envoy" pod "649f1900-0d41-45c8-9cee-e1e6c7fd2610" (UID: "649f1900-0d41-45c8-9cee-e1e6c7fd2610"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:30.385196 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.385157 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:30.385196 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.385190 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkrmj\" (UniqueName: \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-kube-api-access-bkrmj\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:30.385196 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.385204 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:30.385441 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.385215 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/649f1900-0d41-45c8-9cee-e1e6c7fd2610-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:30.828587 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:30.828554 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n"] Oct 30 17:07:31.140083 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:31.139996 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" event={"ID":"649f1900-0d41-45c8-9cee-e1e6c7fd2610","Type":"ContainerDied","Data":"8b08898b258acfb9fd68f7598d6114ff8ab62c9f03919da488f750389c948ed0"} Oct 30 17:07:31.140564 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:31.140105 6400 scope.go:122] "RemoveContainer" containerID="98afc09d83e03534b36ab82bb94d70b5fd35667a5ef2d3817d7c57b9c089f611" Oct 30 17:07:31.140564 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:31.140343 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm" Oct 30 17:07:31.205529 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:31.205496 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm"] Oct 30 17:07:31.216815 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:31.216782 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-967fac-openshift-default-dffc67487-94npm"] Oct 30 17:07:32.969120 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:32.969085 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" path="/var/lib/kubelet/pods/649f1900-0d41-45c8-9cee-e1e6c7fd2610/volumes" Oct 30 17:07:42.267558 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.267514 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" podUID="09583221-74c8-4466-a342-2910378ff059" containerName="main" containerID="cri-o://5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee" gracePeriod=45 Oct 30 17:07:42.571187 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.571163 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:07:42.683997 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.683839 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-tmp-dir\") pod \"09583221-74c8-4466-a342-2910378ff059\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " Oct 30 17:07:42.683997 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.683922 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-dshm\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-dshm\") pod \"09583221-74c8-4466-a342-2910378ff059\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " Oct 30 17:07:42.684260 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.684007 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09583221-74c8-4466-a342-2910378ff059-tls-certs\" (UniqueName: \"kubernetes.io/secret/09583221-74c8-4466-a342-2910378ff059-tls-certs\") pod \"09583221-74c8-4466-a342-2910378ff059\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " Oct 30 17:07:42.684260 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.684091 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-model-cache\") pod \"09583221-74c8-4466-a342-2910378ff059\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " Oct 30 17:07:42.684260 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.684161 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-tmp-dir" pod "09583221-74c8-4466-a342-2910378ff059" (UID: "09583221-74c8-4466-a342-2910378ff059"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:42.684260 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.684171 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-home\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-home\") pod \"09583221-74c8-4466-a342-2910378ff059\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " Oct 30 17:07:42.684451 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.684372 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09583221-74c8-4466-a342-2910378ff059-kube-api-access-mjqnm\" (UniqueName: \"kubernetes.io/projected/09583221-74c8-4466-a342-2910378ff059-kube-api-access-mjqnm\") pod \"09583221-74c8-4466-a342-2910378ff059\" (UID: \"09583221-74c8-4466-a342-2910378ff059\") " Oct 30 17:07:42.684588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.684525 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-home" pod "09583221-74c8-4466-a342-2910378ff059" (UID: "09583221-74c8-4466-a342-2910378ff059"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:42.685454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.685429 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:42.685555 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.685458 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:42.685555 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.685483 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-model-cache" pod "09583221-74c8-4466-a342-2910378ff059" (UID: "09583221-74c8-4466-a342-2910378ff059"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:42.686454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.686437 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-dshm" pod "09583221-74c8-4466-a342-2910378ff059" (UID: "09583221-74c8-4466-a342-2910378ff059"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:42.686656 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.686590 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09583221-74c8-4466-a342-2910378ff059-tls-certs" pod "09583221-74c8-4466-a342-2910378ff059" (UID: "09583221-74c8-4466-a342-2910378ff059"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:42.686880 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.686863 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09583221-74c8-4466-a342-2910378ff059-kube-api-access-mjqnm" pod "09583221-74c8-4466-a342-2910378ff059" (UID: "09583221-74c8-4466-a342-2910378ff059"). InnerVolumeSpecName "kube-api-access-mjqnm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:42.786514 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.786479 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:42.786514 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.786507 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09583221-74c8-4466-a342-2910378ff059-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:42.786514 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.786516 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09583221-74c8-4466-a342-2910378ff059-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:42.786809 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:42.786525 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjqnm\" (UniqueName: \"kubernetes.io/projected/09583221-74c8-4466-a342-2910378ff059-kube-api-access-mjqnm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:43.233583 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:43.233545 6400 generic.go:356] "Generic (PLEG): container finished" podID="09583221-74c8-4466-a342-2910378ff059" containerID="5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee" exitCode=0 Oct 30 17:07:43.233807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:43.233627 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" Oct 30 17:07:43.233807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:43.233629 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" event={"ID":"09583221-74c8-4466-a342-2910378ff059","Type":"ContainerDied","Data":"5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee"} Oct 30 17:07:43.233807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:43.233680 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9" event={"ID":"09583221-74c8-4466-a342-2910378ff059","Type":"ContainerDied","Data":"c19d9457a2e0918cab2c13f3b400ec5095b8d3b19f98d2610116ced9b8b6ee6d"} Oct 30 17:07:43.233807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:43.233699 6400 scope.go:122] "RemoveContainer" containerID="5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee" Oct 30 17:07:43.244520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:43.244502 6400 scope.go:122] "RemoveContainer" containerID="5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee" Oct 30 17:07:43.244913 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:07:43.244884 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee\": container with ID starting with 5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee not found: ID does not exist" containerID="5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee" Oct 30 17:07:43.244992 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:43.244921 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee"} err="failed to get container status \"5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee\": rpc error: code = NotFound desc = could not find container \"5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee\": container with ID starting with 5ecefbacbaf49c4d222781e60f638aef0d0e6bd89fbe536cffa9bb07f699d9ee not found: ID does not exist" Oct 30 17:07:43.283477 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:43.283432 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9"] Oct 30 17:07:43.294685 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:43.294644 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-dc74d5/rate-test-model-dc74d5-kserve-5759bc457c-864j9"] Oct 30 17:07:44.970341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:44.970306 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09583221-74c8-4466-a342-2910378ff059" path="/var/lib/kubelet/pods/09583221-74c8-4466-a342-2910378ff059/volumes" Oct 30 17:07:45.852116 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:45.852073 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" podUID="8431187c-9a0c-4ea5-854a-b4496f5cd277" containerName="main" containerID="cri-o://8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43" gracePeriod=45 Oct 30 17:07:46.152969 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.152944 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:07:46.242283 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.242250 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-tmp-dir\") pod \"8431187c-9a0c-4ea5-854a-b4496f5cd277\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " Oct 30 17:07:46.242454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.242360 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8431187c-9a0c-4ea5-854a-b4496f5cd277-kube-api-access-rts44\" (UniqueName: \"kubernetes.io/projected/8431187c-9a0c-4ea5-854a-b4496f5cd277-kube-api-access-rts44\") pod \"8431187c-9a0c-4ea5-854a-b4496f5cd277\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " Oct 30 17:07:46.242454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.242410 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-dshm\") pod \"8431187c-9a0c-4ea5-854a-b4496f5cd277\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " Oct 30 17:07:46.242532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.242463 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8431187c-9a0c-4ea5-854a-b4496f5cd277-tls-certs\" (UniqueName: \"kubernetes.io/secret/8431187c-9a0c-4ea5-854a-b4496f5cd277-tls-certs\") pod \"8431187c-9a0c-4ea5-854a-b4496f5cd277\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " Oct 30 17:07:46.242532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.242516 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-home\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-home\") pod \"8431187c-9a0c-4ea5-854a-b4496f5cd277\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " Oct 30 17:07:46.242686 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.242613 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-model-cache\") pod \"8431187c-9a0c-4ea5-854a-b4496f5cd277\" (UID: \"8431187c-9a0c-4ea5-854a-b4496f5cd277\") " Oct 30 17:07:46.242686 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.242626 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-tmp-dir" pod "8431187c-9a0c-4ea5-854a-b4496f5cd277" (UID: "8431187c-9a0c-4ea5-854a-b4496f5cd277"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:46.242994 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.242883 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-home" pod "8431187c-9a0c-4ea5-854a-b4496f5cd277" (UID: "8431187c-9a0c-4ea5-854a-b4496f5cd277"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:46.242994 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.242964 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-model-cache" pod "8431187c-9a0c-4ea5-854a-b4496f5cd277" (UID: "8431187c-9a0c-4ea5-854a-b4496f5cd277"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:46.243419 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.243390 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:46.243419 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.243418 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:46.243585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.243430 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:46.245109 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.245090 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8431187c-9a0c-4ea5-854a-b4496f5cd277-tls-certs" pod "8431187c-9a0c-4ea5-854a-b4496f5cd277" (UID: "8431187c-9a0c-4ea5-854a-b4496f5cd277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:07:46.245109 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.245095 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-dshm" pod "8431187c-9a0c-4ea5-854a-b4496f5cd277" (UID: "8431187c-9a0c-4ea5-854a-b4496f5cd277"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:07:46.245218 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.245102 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8431187c-9a0c-4ea5-854a-b4496f5cd277-kube-api-access-rts44" pod "8431187c-9a0c-4ea5-854a-b4496f5cd277" (UID: "8431187c-9a0c-4ea5-854a-b4496f5cd277"). InnerVolumeSpecName "kube-api-access-rts44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:07:46.258802 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.258776 6400 generic.go:356] "Generic (PLEG): container finished" podID="8431187c-9a0c-4ea5-854a-b4496f5cd277" containerID="8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43" exitCode=0 Oct 30 17:07:46.258907 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.258829 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" Oct 30 17:07:46.258907 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.258866 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" event={"ID":"8431187c-9a0c-4ea5-854a-b4496f5cd277","Type":"ContainerDied","Data":"8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43"} Oct 30 17:07:46.258982 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.258912 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n" event={"ID":"8431187c-9a0c-4ea5-854a-b4496f5cd277","Type":"ContainerDied","Data":"181cd0bb48c6dc011656f72f87c01a669e5bb3c3b7c7da4e5e76e415f33c67c0"} Oct 30 17:07:46.258982 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.258933 6400 scope.go:122] "RemoveContainer" containerID="8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43" Oct 30 17:07:46.280427 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.280405 6400 scope.go:122] "RemoveContainer" containerID="8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43" Oct 30 17:07:46.280742 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:07:46.280715 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43\": container with ID starting with 8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43 not found: ID does not exist" containerID="8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43" Oct 30 17:07:46.280810 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.280753 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43"} err="failed to get container status \"8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43\": rpc error: code = NotFound desc = could not find container \"8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43\": container with ID starting with 8d24a77faa9f4b8acc38eddfde81feb527a986f30dfdc43a12c6027c61cdfa43 not found: ID does not exist" Oct 30 17:07:46.315340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.315314 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n"] Oct 30 17:07:46.326228 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.326202 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w1-b4214a/auth-test-model-b4214a-kserve-5499bc7957-5zd6n"] Oct 30 17:07:46.344520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.344492 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rts44\" (UniqueName: \"kubernetes.io/projected/8431187c-9a0c-4ea5-854a-b4496f5cd277-kube-api-access-rts44\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:46.344520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.344516 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8431187c-9a0c-4ea5-854a-b4496f5cd277-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:46.344712 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.344527 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8431187c-9a0c-4ea5-854a-b4496f5cd277-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:07:46.969836 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:46.969800 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8431187c-9a0c-4ea5-854a-b4496f5cd277" path="/var/lib/kubelet/pods/8431187c-9a0c-4ea5-854a-b4496f5cd277/volumes" Oct 30 17:07:47.262636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.262596 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd"] Oct 30 17:07:47.263292 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.263276 6400 cpu_manager.go:414] "containerMap: removing container" podUID="09583221-74c8-4466-a342-2910378ff059" containerName="main" Oct 30 17:07:47.263292 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.263294 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09583221-74c8-4466-a342-2910378ff059" containerName="main" Oct 30 17:07:47.263458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.263305 6400 cpu_manager.go:414] "containerMap: removing container" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" Oct 30 17:07:47.263458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.263310 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" Oct 30 17:07:47.263458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.263320 6400 cpu_manager.go:414] "containerMap: removing container" podUID="8431187c-9a0c-4ea5-854a-b4496f5cd277" containerName="main" Oct 30 17:07:47.263458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.263325 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8431187c-9a0c-4ea5-854a-b4496f5cd277" containerName="main" Oct 30 17:07:47.263458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.263453 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="09583221-74c8-4466-a342-2910378ff059" containerName="main" Oct 30 17:07:47.263458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.263460 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="649f1900-0d41-45c8-9cee-e1e6c7fd2610" containerName="istio-proxy" Oct 30 17:07:47.263719 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.263471 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="8431187c-9a0c-4ea5-854a-b4496f5cd277" containerName="main" Oct 30 17:07:47.267503 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.267478 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.301684 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.301650 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd"] Oct 30 17:07:47.302404 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.302380 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-e1e830-serving-cert\"" Oct 30 17:07:47.358771 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.358740 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj2wh\" (UniqueName: \"kubernetes.io/projected/a522dd46-312d-47ac-8a11-07a76546094a-kube-api-access-vj2wh\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.358920 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.358806 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.358977 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.358926 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a522dd46-312d-47ac-8a11-07a76546094a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.359147 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.359130 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-metrics-certs\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.460285 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.460248 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj2wh\" (UniqueName: \"kubernetes.io/projected/a522dd46-312d-47ac-8a11-07a76546094a-kube-api-access-vj2wh\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.460458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.460312 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.460458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.460361 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a522dd46-312d-47ac-8a11-07a76546094a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.460614 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.460579 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-metrics-certs\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.461134 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.461107 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a522dd46-312d-47ac-8a11-07a76546094a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.462908 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.462881 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.463172 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.463149 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-metrics-certs\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.482992 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.482966 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj2wh\" (UniqueName: \"kubernetes.io/projected/a522dd46-312d-47ac-8a11-07a76546094a-kube-api-access-vj2wh\") pod \"maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.581213 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.581123 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:47.891506 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:47.891479 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd"] Oct 30 17:07:47.892882 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:07:47.892853 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda522dd46_312d_47ac_8a11_07a76546094a.slice/crio-4c31e27753902992d3da3e478884cae08932d1ead966ef23754da3f4fc73a13e WatchSource:0}: Error finding container 4c31e27753902992d3da3e478884cae08932d1ead966ef23754da3f4fc73a13e: Status 404 returned error can't find the container with id 4c31e27753902992d3da3e478884cae08932d1ead966ef23754da3f4fc73a13e Oct 30 17:07:48.278768 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:48.278724 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" event={"ID":"a522dd46-312d-47ac-8a11-07a76546094a","Type":"ContainerStarted","Data":"4c31e27753902992d3da3e478884cae08932d1ead966ef23754da3f4fc73a13e"} Oct 30 17:07:49.287400 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:49.287364 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" event={"ID":"a522dd46-312d-47ac-8a11-07a76546094a","Type":"ContainerStarted","Data":"19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294"} Oct 30 17:07:49.287818 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:49.287433 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:49.331124 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:49.331071 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" podStartSLOduration=1.964965704 podStartE2EDuration="2.331052524s" totalImagesPullingTime="366.08682ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:07:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:07:49.327409358 +0000 UTC m=+2778.477690360" watchObservedRunningTime="2026-10-30 17:07:49.331052524 +0000 UTC m=+2778.481333512" Oct 30 17:07:54.186153 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:54.186114 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54"] Oct 30 17:07:54.610333 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:07:54.610290 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:07:55.301473 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:55.301442 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:07:58.801841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:58.801806 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q"] Oct 30 17:07:58.802663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:07:58.802596 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" containerID="cri-o://b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e" gracePeriod=30 Oct 30 17:08:04.103162 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.103132 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 17:08:04.145225 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145191 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-kube-api-access-m6zn4\" (UniqueName: \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-kube-api-access-m6zn4\") pod \"19c3c768-317f-4692-8e4e-27fe738d96cc\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " Oct 30 17:08:04.145394 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145275 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19c3c768-317f-4692-8e4e-27fe738d96cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c3c768-317f-4692-8e4e-27fe738d96cc-istiod-ca-cert\") pod \"19c3c768-317f-4692-8e4e-27fe738d96cc\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " Oct 30 17:08:04.145438 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145404 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-token\" (UniqueName: \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-token\") pod \"19c3c768-317f-4692-8e4e-27fe738d96cc\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " Oct 30 17:08:04.145474 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145461 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-certs\") pod \"19c3c768-317f-4692-8e4e-27fe738d96cc\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " Oct 30 17:08:04.145644 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145577 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-credential-socket\") pod \"19c3c768-317f-4692-8e4e-27fe738d96cc\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " Oct 30 17:08:04.145828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145665 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-data\") pod \"19c3c768-317f-4692-8e4e-27fe738d96cc\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " Oct 30 17:08:04.145828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145802 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-podinfo\") pod \"19c3c768-317f-4692-8e4e-27fe738d96cc\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " Oct 30 17:08:04.145956 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145835 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-envoy\") pod \"19c3c768-317f-4692-8e4e-27fe738d96cc\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " Oct 30 17:08:04.145956 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145925 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-socket\") pod \"19c3c768-317f-4692-8e4e-27fe738d96cc\" (UID: \"19c3c768-317f-4692-8e4e-27fe738d96cc\") " Oct 30 17:08:04.146230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145828 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-certs" pod "19c3c768-317f-4692-8e4e-27fe738d96cc" (UID: "19c3c768-317f-4692-8e4e-27fe738d96cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:04.146230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145952 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-data" pod "19c3c768-317f-4692-8e4e-27fe738d96cc" (UID: "19c3c768-317f-4692-8e4e-27fe738d96cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:04.146230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145838 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19c3c768-317f-4692-8e4e-27fe738d96cc-istiod-ca-cert" pod "19c3c768-317f-4692-8e4e-27fe738d96cc" (UID: "19c3c768-317f-4692-8e4e-27fe738d96cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:04.146230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.145843 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-credential-socket" pod "19c3c768-317f-4692-8e4e-27fe738d96cc" (UID: "19c3c768-317f-4692-8e4e-27fe738d96cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:04.146457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.146252 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-socket" pod "19c3c768-317f-4692-8e4e-27fe738d96cc" (UID: "19c3c768-317f-4692-8e4e-27fe738d96cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:04.146552 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.146534 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.146660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.146553 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19c3c768-317f-4692-8e4e-27fe738d96cc-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.146660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.146564 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.146660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.146572 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.146660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.146580 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.147925 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.147901 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-token" pod "19c3c768-317f-4692-8e4e-27fe738d96cc" (UID: "19c3c768-317f-4692-8e4e-27fe738d96cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:04.148018 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.147936 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-kube-api-access-m6zn4" pod "19c3c768-317f-4692-8e4e-27fe738d96cc" (UID: "19c3c768-317f-4692-8e4e-27fe738d96cc"). InnerVolumeSpecName "kube-api-access-m6zn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:04.148457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.148429 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-podinfo" pod "19c3c768-317f-4692-8e4e-27fe738d96cc" (UID: "19c3c768-317f-4692-8e4e-27fe738d96cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:08:04.148554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.148453 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-envoy" pod "19c3c768-317f-4692-8e4e-27fe738d96cc" (UID: "19c3c768-317f-4692-8e4e-27fe738d96cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:04.248341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.248306 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.248341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.248336 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.248341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.248345 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19c3c768-317f-4692-8e4e-27fe738d96cc-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.248589 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.248354 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m6zn4\" (UniqueName: \"kubernetes.io/projected/19c3c768-317f-4692-8e4e-27fe738d96cc-kube-api-access-m6zn4\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:04.438127 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.438021 6400 generic.go:356] "Generic (PLEG): container finished" podID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerID="b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e" exitCode=0 Oct 30 17:08:04.438127 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.438110 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" Oct 30 17:08:04.438353 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.438107 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" event={"ID":"19c3c768-317f-4692-8e4e-27fe738d96cc","Type":"ContainerDied","Data":"b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e"} Oct 30 17:08:04.438353 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.438262 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q" event={"ID":"19c3c768-317f-4692-8e4e-27fe738d96cc","Type":"ContainerDied","Data":"15059e106c4beb8719721980e7f6b7dc22c9878b67ac17818f6bf7ed2b0618cf"} Oct 30 17:08:04.438353 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.438300 6400 scope.go:122] "RemoveContainer" containerID="b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e" Oct 30 17:08:04.451117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.451072 6400 scope.go:122] "RemoveContainer" containerID="01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a" Oct 30 17:08:04.466016 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.463996 6400 scope.go:122] "RemoveContainer" containerID="b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e" Oct 30 17:08:04.466266 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:08:04.466235 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e\": container with ID starting with b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e not found: ID does not exist" containerID="b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e" Oct 30 17:08:04.466364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.466278 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e"} err="failed to get container status \"b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e\": rpc error: code = NotFound desc = could not find container \"b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e\": container with ID starting with b48a3fb3c20b10da545f4adb0124212022b116d8b4e8cf59546b64a749260f8e not found: ID does not exist" Oct 30 17:08:04.466364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.466298 6400 scope.go:122] "RemoveContainer" containerID="01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a" Oct 30 17:08:04.466848 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:08:04.466813 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a\": container with ID starting with 01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a not found: ID does not exist" containerID="01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a" Oct 30 17:08:04.466954 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.466855 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a"} err="failed to get container status \"01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a\": rpc error: code = NotFound desc = could not find container \"01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a\": container with ID starting with 01654fdbce7aaca3a04a018ba41c2763980c86c8e603441961f8d126a8ad7a7a not found: ID does not exist" Oct 30 17:08:04.490416 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.490391 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q"] Oct 30 17:08:04.503382 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.503351 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-830ba9-openshift-default-6bb89896cc-6rr4q"] Oct 30 17:08:04.984217 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:04.984174 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" path="/var/lib/kubelet/pods/19c3c768-317f-4692-8e4e-27fe738d96cc/volumes" Oct 30 17:08:06.166636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.166568 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk"] Oct 30 17:08:06.167364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.167344 6400 cpu_manager.go:414] "containerMap: removing container" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" Oct 30 17:08:06.167413 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.167369 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" Oct 30 17:08:06.167652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.167624 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" Oct 30 17:08:06.172006 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.171988 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:08:06.187678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.187657 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-55e83e\"" Oct 30 17:08:06.198729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.198707 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-55e83e-dockercfg-jhwmc\"" Oct 30 17:08:06.202551 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.202525 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk"] Oct 30 17:08:06.302261 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.302215 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133c0bcd-04a0-410e-aafc-73c4279bd13f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk\" (UID: \"133c0bcd-04a0-410e-aafc-73c4279bd13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:08:06.302484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.302459 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt4r6\" (UniqueName: \"kubernetes.io/projected/133c0bcd-04a0-410e-aafc-73c4279bd13f-kube-api-access-lt4r6\") pod \"payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk\" (UID: \"133c0bcd-04a0-410e-aafc-73c4279bd13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:08:06.404388 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.404354 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt4r6\" (UniqueName: \"kubernetes.io/projected/133c0bcd-04a0-410e-aafc-73c4279bd13f-kube-api-access-lt4r6\") pod \"payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk\" (UID: \"133c0bcd-04a0-410e-aafc-73c4279bd13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:08:06.404679 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.404510 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133c0bcd-04a0-410e-aafc-73c4279bd13f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk\" (UID: \"133c0bcd-04a0-410e-aafc-73c4279bd13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:08:06.405351 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.405331 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133c0bcd-04a0-410e-aafc-73c4279bd13f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk\" (UID: \"133c0bcd-04a0-410e-aafc-73c4279bd13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:08:06.427643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.427546 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt4r6\" (UniqueName: \"kubernetes.io/projected/133c0bcd-04a0-410e-aafc-73c4279bd13f-kube-api-access-lt4r6\") pod \"payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk\" (UID: \"133c0bcd-04a0-410e-aafc-73c4279bd13f\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:08:06.486329 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.486292 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:08:06.796160 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:06.796129 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk"] Oct 30 17:08:06.796987 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:08:06.796961 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod133c0bcd_04a0_410e_aafc_73c4279bd13f.slice/crio-5984eca59aa4fd228804ca388f1ff537f3ee62d376aa15856c7283119e92ec48 WatchSource:0}: Error finding container 5984eca59aa4fd228804ca388f1ff537f3ee62d376aa15856c7283119e92ec48: Status 404 returned error can't find the container with id 5984eca59aa4fd228804ca388f1ff537f3ee62d376aa15856c7283119e92ec48 Oct 30 17:08:07.464902 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:07.464850 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" event={"ID":"133c0bcd-04a0-410e-aafc-73c4279bd13f","Type":"ContainerStarted","Data":"f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e"} Oct 30 17:08:07.465399 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:07.464929 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" event={"ID":"133c0bcd-04a0-410e-aafc-73c4279bd13f","Type":"ContainerStarted","Data":"5984eca59aa4fd228804ca388f1ff537f3ee62d376aa15856c7283119e92ec48"} Oct 30 17:08:07.465399 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:07.464958 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:08:07.505845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:07.505787 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" podStartSLOduration=1.5057728049999999 podStartE2EDuration="1.505772805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:07.504405251 +0000 UTC m=+2796.654686241" watchObservedRunningTime="2026-10-30 17:08:07.505772805 +0000 UTC m=+2796.656053818" Oct 30 17:08:09.210453 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.210412 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" podUID="88d766ab-40c9-4c3f-be73-b3d4f4d933b5" containerName="main" containerID="cri-o://162c9ac6afbeba031a7b12bfef141daede487cbec0f0d9bd8f46ce71f48aa132" gracePeriod=45 Oct 30 17:08:09.480389 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.480361 6400 generic.go:356] "Generic (PLEG): container finished" podID="88d766ab-40c9-4c3f-be73-b3d4f4d933b5" containerID="162c9ac6afbeba031a7b12bfef141daede487cbec0f0d9bd8f46ce71f48aa132" exitCode=0 Oct 30 17:08:09.480542 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.480433 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" event={"ID":"88d766ab-40c9-4c3f-be73-b3d4f4d933b5","Type":"ContainerDied","Data":"162c9ac6afbeba031a7b12bfef141daede487cbec0f0d9bd8f46ce71f48aa132"} Oct 30 17:08:09.480542 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.480489 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" event={"ID":"88d766ab-40c9-4c3f-be73-b3d4f4d933b5","Type":"ContainerDied","Data":"8fb33c345144b2613c9d2ff0994779b2848aa54a842a837ec39ff9d6ef1379e7"} Oct 30 17:08:09.480542 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.480510 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fb33c345144b2613c9d2ff0994779b2848aa54a842a837ec39ff9d6ef1379e7" Oct 30 17:08:09.505171 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.505144 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:08:09.650966 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.650926 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-model-cache\") pod \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " Oct 30 17:08:09.651181 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.651096 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tmp-dir\") pod \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " Oct 30 17:08:09.651181 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.651155 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-home\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-home\") pod \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " Oct 30 17:08:09.651311 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.651194 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-dshm\") pod \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " Oct 30 17:08:09.651311 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.651221 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-kube-api-access-85rxs\" (UniqueName: \"kubernetes.io/projected/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-kube-api-access-85rxs\") pod \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " Oct 30 17:08:09.651311 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.651270 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-model-cache" pod "88d766ab-40c9-4c3f-be73-b3d4f4d933b5" (UID: "88d766ab-40c9-4c3f-be73-b3d4f4d933b5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:09.651465 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.651391 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tls-certs\" (UniqueName: \"kubernetes.io/secret/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tls-certs\") pod \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\" (UID: \"88d766ab-40c9-4c3f-be73-b3d4f4d933b5\") " Oct 30 17:08:09.651526 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.651386 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-home" pod "88d766ab-40c9-4c3f-be73-b3d4f4d933b5" (UID: "88d766ab-40c9-4c3f-be73-b3d4f4d933b5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:09.651962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.651937 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tmp-dir" pod "88d766ab-40c9-4c3f-be73-b3d4f4d933b5" (UID: "88d766ab-40c9-4c3f-be73-b3d4f4d933b5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:09.652243 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.652216 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:09.652243 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.652245 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:09.652243 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.652258 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:09.653941 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.653904 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-dshm" pod "88d766ab-40c9-4c3f-be73-b3d4f4d933b5" (UID: "88d766ab-40c9-4c3f-be73-b3d4f4d933b5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:09.654028 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.653993 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tls-certs" pod "88d766ab-40c9-4c3f-be73-b3d4f4d933b5" (UID: "88d766ab-40c9-4c3f-be73-b3d4f4d933b5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:09.654176 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.654153 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-kube-api-access-85rxs" pod "88d766ab-40c9-4c3f-be73-b3d4f4d933b5" (UID: "88d766ab-40c9-4c3f-be73-b3d4f4d933b5"). InnerVolumeSpecName "kube-api-access-85rxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:09.754058 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.754031 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:09.754058 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.754058 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85rxs\" (UniqueName: \"kubernetes.io/projected/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-kube-api-access-85rxs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:09.754270 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:09.754069 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88d766ab-40c9-4c3f-be73-b3d4f4d933b5-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:10.495258 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:10.495217 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54" Oct 30 17:08:10.546668 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:10.546633 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54"] Oct 30 17:08:10.557688 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:10.557660 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w1-ae5afd/auth-test-model-ae5afd-kserve-76dff69645-w8b54"] Oct 30 17:08:10.969665 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:10.969556 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88d766ab-40c9-4c3f-be73-b3d4f4d933b5" path="/var/lib/kubelet/pods/88d766ab-40c9-4c3f-be73-b3d4f4d933b5/volumes" Oct 30 17:08:18.474520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:18.474490 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:08:24.601171 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:08:24.601134 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:08:32.014828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.014787 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj"] Oct 30 17:08:32.015371 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.015045 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" containerID="cri-o://b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506" gracePeriod=30 Oct 30 17:08:32.107562 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.107516 6400 scope.go:122] "RemoveContainer" containerID="22bb7915341b5d298cb18f298c2b5789e2e1c650ea584515044e162d07b3ca2d" Oct 30 17:08:32.552123 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.552089 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62"] Oct 30 17:08:32.552815 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.552797 6400 cpu_manager.go:414] "containerMap: removing container" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" Oct 30 17:08:32.552865 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.552817 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" Oct 30 17:08:32.552865 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.552854 6400 cpu_manager.go:414] "containerMap: removing container" podUID="88d766ab-40c9-4c3f-be73-b3d4f4d933b5" containerName="main" Oct 30 17:08:32.552865 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.552861 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88d766ab-40c9-4c3f-be73-b3d4f4d933b5" containerName="main" Oct 30 17:08:32.552983 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.552973 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="88d766ab-40c9-4c3f-be73-b3d4f4d933b5" containerName="main" Oct 30 17:08:32.553014 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.552988 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="19c3c768-317f-4692-8e4e-27fe738d96cc" containerName="istio-proxy" Oct 30 17:08:32.556577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.556557 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.571651 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.571576 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-e4048c-serving-cert\"" Oct 30 17:08:32.573650 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.573586 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62"] Oct 30 17:08:32.654864 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.654813 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.655072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.654965 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-metrics-certs\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.655072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.655033 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2spws\" (UniqueName: \"kubernetes.io/projected/a8566fdd-4890-45e6-8654-d4ba5a0a9017-kube-api-access-2spws\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.655188 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.655090 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8566fdd-4890-45e6-8654-d4ba5a0a9017-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.756848 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.756813 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-metrics-certs\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.757037 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.756894 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2spws\" (UniqueName: \"kubernetes.io/projected/a8566fdd-4890-45e6-8654-d4ba5a0a9017-kube-api-access-2spws\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.757037 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.756930 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8566fdd-4890-45e6-8654-d4ba5a0a9017-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.757207 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.757073 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.757736 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.757704 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8566fdd-4890-45e6-8654-d4ba5a0a9017-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.759664 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.759595 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-maas-api-tls\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.759776 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.759747 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-metrics-certs\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.779577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.779553 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2spws\" (UniqueName: \"kubernetes.io/projected/a8566fdd-4890-45e6-8654-d4ba5a0a9017-kube-api-access-2spws\") pod \"maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:32.870646 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:32.870476 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:33.182256 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:08:33.182201 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8566fdd_4890_45e6_8654_d4ba5a0a9017.slice/crio-c01e183794a45dda0bbbe08883f846ab7f9f51992cfb3f1922dc85bdb84dfab4 WatchSource:0}: Error finding container c01e183794a45dda0bbbe08883f846ab7f9f51992cfb3f1922dc85bdb84dfab4: Status 404 returned error can't find the container with id c01e183794a45dda0bbbe08883f846ab7f9f51992cfb3f1922dc85bdb84dfab4 Oct 30 17:08:33.183258 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:33.183229 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62"] Oct 30 17:08:33.183873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:33.183854 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:08:33.308423 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:33.308378 6400 patch_prober.go:29] interesting pod/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:08:33.308616 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:33.308433 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:08:33.692910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:33.692881 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" event={"ID":"a8566fdd-4890-45e6-8654-d4ba5a0a9017","Type":"ContainerStarted","Data":"c01e183794a45dda0bbbe08883f846ab7f9f51992cfb3f1922dc85bdb84dfab4"} Oct 30 17:08:34.706665 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:34.706629 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" event={"ID":"a8566fdd-4890-45e6-8654-d4ba5a0a9017","Type":"ContainerStarted","Data":"cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f"} Oct 30 17:08:34.707215 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:34.706820 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:34.749756 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:34.749709 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" podStartSLOduration=2.287636711 podStartE2EDuration="2.74969625s" totalImagesPullingTime="462.059539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:34.746277752 +0000 UTC m=+2823.896558741" watchObservedRunningTime="2026-10-30 17:08:34.74969625 +0000 UTC m=+2823.899977246" Oct 30 17:08:37.296096 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.296032 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 17:08:37.433532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.433490 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-certs\") pod \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " Oct 30 17:08:37.433788 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.433551 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-token\") pod \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " Oct 30 17:08:37.433788 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.433657 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-data\") pod \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " Oct 30 17:08:37.433788 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.433706 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-kube-api-access-g8t2t\" (UniqueName: \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-kube-api-access-g8t2t\") pod \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " Oct 30 17:08:37.433962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.433819 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-envoy\") pod \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " Oct 30 17:08:37.433962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.433856 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-certs" pod "fc4a7872-45cb-498b-a8a4-92eeef9c76b7" (UID: "fc4a7872-45cb-498b-a8a4-92eeef9c76b7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:37.433962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.433866 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-podinfo\") pod \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " Oct 30 17:08:37.434128 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.433969 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istiod-ca-cert\") pod \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " Oct 30 17:08:37.434128 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.434075 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-credential-socket\") pod \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " Oct 30 17:08:37.434237 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.434172 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-socket\") pod \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\" (UID: \"fc4a7872-45cb-498b-a8a4-92eeef9c76b7\") " Oct 30 17:08:37.434513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.434469 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-credential-socket" pod "fc4a7872-45cb-498b-a8a4-92eeef9c76b7" (UID: "fc4a7872-45cb-498b-a8a4-92eeef9c76b7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:37.434513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.434490 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-data" pod "fc4a7872-45cb-498b-a8a4-92eeef9c76b7" (UID: "fc4a7872-45cb-498b-a8a4-92eeef9c76b7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:37.434803 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.434581 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istiod-ca-cert" pod "fc4a7872-45cb-498b-a8a4-92eeef9c76b7" (UID: "fc4a7872-45cb-498b-a8a4-92eeef9c76b7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:08:37.434803 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.434708 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-socket" pod "fc4a7872-45cb-498b-a8a4-92eeef9c76b7" (UID: "fc4a7872-45cb-498b-a8a4-92eeef9c76b7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:37.435450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.435338 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.435450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.435365 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.435450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.435381 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.435450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.435397 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.435450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.435411 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.436585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.436548 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-podinfo" pod "fc4a7872-45cb-498b-a8a4-92eeef9c76b7" (UID: "fc4a7872-45cb-498b-a8a4-92eeef9c76b7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:08:37.436889 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.436862 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-kube-api-access-g8t2t" pod "fc4a7872-45cb-498b-a8a4-92eeef9c76b7" (UID: "fc4a7872-45cb-498b-a8a4-92eeef9c76b7"). InnerVolumeSpecName "kube-api-access-g8t2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:37.437171 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.437154 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-token" pod "fc4a7872-45cb-498b-a8a4-92eeef9c76b7" (UID: "fc4a7872-45cb-498b-a8a4-92eeef9c76b7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:37.437404 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.437387 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-envoy" pod "fc4a7872-45cb-498b-a8a4-92eeef9c76b7" (UID: "fc4a7872-45cb-498b-a8a4-92eeef9c76b7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:37.500581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.500547 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp"] Oct 30 17:08:37.537372 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.537339 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.537372 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.537366 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8t2t\" (UniqueName: \"kubernetes.io/projected/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-kube-api-access-g8t2t\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.537372 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.537377 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.537624 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.537385 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fc4a7872-45cb-498b-a8a4-92eeef9c76b7-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:37.731130 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.731025 6400 generic.go:356] "Generic (PLEG): container finished" podID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerID="b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506" exitCode=0 Oct 30 17:08:37.731130 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.731092 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" Oct 30 17:08:37.731130 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.731113 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" event={"ID":"fc4a7872-45cb-498b-a8a4-92eeef9c76b7","Type":"ContainerDied","Data":"b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506"} Oct 30 17:08:37.731444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.731186 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj" event={"ID":"fc4a7872-45cb-498b-a8a4-92eeef9c76b7","Type":"ContainerDied","Data":"9a6b5ca5f9858073b4fe9c24cc837a02a881ec96f4dc45b2e9cf8e06a42f3b5b"} Oct 30 17:08:37.731444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.731218 6400 scope.go:122] "RemoveContainer" containerID="b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506" Oct 30 17:08:37.748100 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.748037 6400 scope.go:122] "RemoveContainer" containerID="2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393" Oct 30 17:08:37.761519 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.761495 6400 scope.go:122] "RemoveContainer" containerID="b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506" Oct 30 17:08:37.761847 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:08:37.761824 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506\": container with ID starting with b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506 not found: ID does not exist" containerID="b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506" Oct 30 17:08:37.761946 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.761860 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506"} err="failed to get container status \"b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506\": rpc error: code = NotFound desc = could not find container \"b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506\": container with ID starting with b90098502bafed59d7d20ac2d7182d81af42903a9bd06a49f5d1b4d0ff358506 not found: ID does not exist" Oct 30 17:08:37.761946 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.761882 6400 scope.go:122] "RemoveContainer" containerID="2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393" Oct 30 17:08:37.762195 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:08:37.762170 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393\": container with ID starting with 2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393 not found: ID does not exist" containerID="2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393" Oct 30 17:08:37.762274 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.762209 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393"} err="failed to get container status \"2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393\": rpc error: code = NotFound desc = could not find container \"2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393\": container with ID starting with 2086f28160651bf1837c0a621555022d6e7d63f040ace6c450ed50557f45d393 not found: ID does not exist" Oct 30 17:08:37.784771 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.784737 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj"] Oct 30 17:08:37.795773 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:37.795744 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w6-e5bfb0-openshift-default-689d9f54b5-gpvgj"] Oct 30 17:08:38.970566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:38.970527 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" path="/var/lib/kubelet/pods/fc4a7872-45cb-498b-a8a4-92eeef9c76b7/volumes" Oct 30 17:08:40.720425 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:40.720394 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:08:51.621216 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.621179 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4"] Oct 30 17:08:51.622018 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.621998 6400 cpu_manager.go:414] "containerMap: removing container" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" Oct 30 17:08:51.622092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.622020 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" Oct 30 17:08:51.622170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.622161 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" Oct 30 17:08:51.622212 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.622174 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" Oct 30 17:08:51.627269 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.627249 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.648848 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.648821 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-4ac125-serving-cert\"" Oct 30 17:08:51.654004 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.653975 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4"] Oct 30 17:08:51.707110 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.707077 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-metrics-certs\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.707304 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.707116 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.707304 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.707282 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65zwh\" (UniqueName: \"kubernetes.io/projected/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-kube-api-access-65zwh\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.707439 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.707423 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.809136 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.809101 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-metrics-certs\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.809136 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.809143 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.809370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.809215 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65zwh\" (UniqueName: \"kubernetes.io/projected/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-kube-api-access-65zwh\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.809370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.809290 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.809975 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.809955 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.811845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.811826 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-maas-api-tls\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.811946 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.811929 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-metrics-certs\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.832546 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.832520 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65zwh\" (UniqueName: \"kubernetes.io/projected/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-kube-api-access-65zwh\") pod \"maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:51.942525 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:51.942451 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:52.252225 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.252190 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4"] Oct 30 17:08:52.253294 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:08:52.253264 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d76b4b_e64a_4e0d_bcac_30a8104e2c20.slice/crio-d77cff7fc16c59309f4f258a1d0d38d5629a8e4e4796790a67867153a95f400e WatchSource:0}: Error finding container d77cff7fc16c59309f4f258a1d0d38d5629a8e4e4796790a67867153a95f400e: Status 404 returned error can't find the container with id d77cff7fc16c59309f4f258a1d0d38d5629a8e4e4796790a67867153a95f400e Oct 30 17:08:52.525265 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.525158 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" containerID="cri-o://47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8" gracePeriod=45 Oct 30 17:08:52.575853 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.575814 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" probeResult="failure" output="Get \"https://10.134.0.79:8000/ready\": dial tcp 10.134.0.79:8000: connect: connection refused" Oct 30 17:08:52.797447 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.797420 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 17:08:52.854347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.854304 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" event={"ID":"90d76b4b-e64a-4e0d-bcac-30a8104e2c20","Type":"ContainerStarted","Data":"d77cff7fc16c59309f4f258a1d0d38d5629a8e4e4796790a67867153a95f400e"} Oct 30 17:08:52.857908 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.857874 6400 generic.go:356] "Generic (PLEG): container finished" podID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerID="47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8" exitCode=0 Oct 30 17:08:52.858461 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.858434 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" event={"ID":"8fc34352-0241-44b1-96fe-ddccd74ac59a","Type":"ContainerDied","Data":"47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8"} Oct 30 17:08:52.858702 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.858507 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" event={"ID":"8fc34352-0241-44b1-96fe-ddccd74ac59a","Type":"ContainerDied","Data":"ad74e0369d77598ca122838fe07a422de26d319d50c9727f8d1b85bddbb3fdcc"} Oct 30 17:08:52.858702 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.858565 6400 scope.go:122] "RemoveContainer" containerID="47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8" Oct 30 17:08:52.858950 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.858744 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp" Oct 30 17:08:52.882578 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.882551 6400 scope.go:122] "RemoveContainer" containerID="47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8" Oct 30 17:08:52.883253 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:08:52.883227 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8\": container with ID starting with 47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8 not found: ID does not exist" containerID="47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8" Oct 30 17:08:52.883426 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.883398 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8"} err="failed to get container status \"47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8\": rpc error: code = NotFound desc = could not find container \"47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8\": container with ID starting with 47e314f6031f8a2da7b67bdcabbe258c6b95d9a3935a0c463e7cb82bedb80da8 not found: ID does not exist" Oct 30 17:08:52.931183 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.931153 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-model-cache\") pod \"8fc34352-0241-44b1-96fe-ddccd74ac59a\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " Oct 30 17:08:52.931350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.931310 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-dshm\") pod \"8fc34352-0241-44b1-96fe-ddccd74ac59a\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " Oct 30 17:08:52.931422 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.931357 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8fc34352-0241-44b1-96fe-ddccd74ac59a-tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc34352-0241-44b1-96fe-ddccd74ac59a-tls-certs\") pod \"8fc34352-0241-44b1-96fe-ddccd74ac59a\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " Oct 30 17:08:52.931422 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.931392 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-tmp-dir\") pod \"8fc34352-0241-44b1-96fe-ddccd74ac59a\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " Oct 30 17:08:52.931529 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.931409 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-model-cache" pod "8fc34352-0241-44b1-96fe-ddccd74ac59a" (UID: "8fc34352-0241-44b1-96fe-ddccd74ac59a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:52.931529 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.931433 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fc34352-0241-44b1-96fe-ddccd74ac59a-kube-api-access-8l4tp\" (UniqueName: \"kubernetes.io/projected/8fc34352-0241-44b1-96fe-ddccd74ac59a-kube-api-access-8l4tp\") pod \"8fc34352-0241-44b1-96fe-ddccd74ac59a\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " Oct 30 17:08:52.931529 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.931490 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-home\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-home\") pod \"8fc34352-0241-44b1-96fe-ddccd74ac59a\" (UID: \"8fc34352-0241-44b1-96fe-ddccd74ac59a\") " Oct 30 17:08:52.931739 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.931697 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-tmp-dir" pod "8fc34352-0241-44b1-96fe-ddccd74ac59a" (UID: "8fc34352-0241-44b1-96fe-ddccd74ac59a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:52.932030 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.932002 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-home" pod "8fc34352-0241-44b1-96fe-ddccd74ac59a" (UID: "8fc34352-0241-44b1-96fe-ddccd74ac59a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:52.932386 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.932358 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:52.932484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.932396 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:52.932484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.932410 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:52.934572 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.934544 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-dshm" pod "8fc34352-0241-44b1-96fe-ddccd74ac59a" (UID: "8fc34352-0241-44b1-96fe-ddccd74ac59a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:08:52.934963 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.934765 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8fc34352-0241-44b1-96fe-ddccd74ac59a-tls-certs" pod "8fc34352-0241-44b1-96fe-ddccd74ac59a" (UID: "8fc34352-0241-44b1-96fe-ddccd74ac59a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:08:52.934963 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:52.934781 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc34352-0241-44b1-96fe-ddccd74ac59a-kube-api-access-8l4tp" pod "8fc34352-0241-44b1-96fe-ddccd74ac59a" (UID: "8fc34352-0241-44b1-96fe-ddccd74ac59a"). InnerVolumeSpecName "kube-api-access-8l4tp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:08:53.034455 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:53.034423 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l4tp\" (UniqueName: \"kubernetes.io/projected/8fc34352-0241-44b1-96fe-ddccd74ac59a-kube-api-access-8l4tp\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.034455 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:53.034451 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8fc34352-0241-44b1-96fe-ddccd74ac59a-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.034669 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:53.034464 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8fc34352-0241-44b1-96fe-ddccd74ac59a-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:08:53.205424 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:53.205343 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp"] Oct 30 17:08:53.209178 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:53.209149 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-d693d0/distinct-w6-d693d0-kserve-5d6ff6b8b9-9w5mp"] Oct 30 17:08:53.867509 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:53.867475 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" event={"ID":"90d76b4b-e64a-4e0d-bcac-30a8104e2c20","Type":"ContainerStarted","Data":"cc1928b53bd13b2bf45c485111bdece3dbfedfb6e86309d6ba0b55adaac4b46c"} Oct 30 17:08:53.868021 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:53.867532 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:53.909907 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:53.909844 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" podStartSLOduration=2.272999567 podStartE2EDuration="2.909828261s" totalImagesPullingTime="636.828694ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:08:53.907542032 +0000 UTC m=+2843.057823020" watchObservedRunningTime="2026-10-30 17:08:53.909828261 +0000 UTC m=+2843.060109247" Oct 30 17:08:54.600745 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:08:54.600712 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:08:54.969008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:54.968932 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" path="/var/lib/kubelet/pods/8fc34352-0241-44b1-96fe-ddccd74ac59a/volumes" Oct 30 17:08:59.866800 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:59.866764 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb"] Oct 30 17:08:59.867700 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:59.867678 6400 cpu_manager.go:414] "containerMap: removing container" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" Oct 30 17:08:59.867700 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:59.867699 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" Oct 30 17:08:59.867882 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:59.867719 6400 cpu_manager.go:414] "containerMap: removing container" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" Oct 30 17:08:59.867882 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:59.867724 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4a7872-45cb-498b-a8a4-92eeef9c76b7" containerName="istio-proxy" Oct 30 17:08:59.867882 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:59.867860 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fc34352-0241-44b1-96fe-ddccd74ac59a" containerName="main" Oct 30 17:08:59.871644 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:59.871595 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:08:59.882508 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:59.882487 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:08:59.886014 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:59.885993 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-bfecdd-openshift-default-dockercfg-2cb9m\"" Oct 30 17:08:59.902182 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:08:59.902159 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb"] Oct 30 17:09:00.045068 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.045038 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-token\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.045368 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.045332 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istiod-ca-cert\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.045460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.045421 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-envoy\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.045587 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.045472 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-certs\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.045679 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.045658 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-data\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.045736 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.045717 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-credential-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.045799 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.045784 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.045849 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.045812 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-podinfo\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.045849 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.045827 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvbpj\" (UniqueName: \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-kube-api-access-nvbpj\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147064 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.146965 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istiod-ca-cert\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147064 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147030 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-envoy\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147064 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147056 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-certs\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147090 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-data\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147107 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-credential-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147151 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147179 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-podinfo\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147205 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvbpj\" (UniqueName: \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-kube-api-access-nvbpj\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147267 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-token\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147742 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147574 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-data\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147742 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147662 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147742 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147694 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-credential-socket\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.147911 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.147751 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-certs\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.148163 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.148104 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istiod-ca-cert\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.149829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.149807 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-envoy\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.150123 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.150101 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-podinfo\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.172359 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.172325 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-token\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.172511 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.172446 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvbpj\" (UniqueName: \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-kube-api-access-nvbpj\") pod \"e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.188027 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.187996 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:00.520123 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.520092 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb"] Oct 30 17:09:00.520766 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:09:00.520740 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6efd7c15_73f9_449c_b4fe_ffbb9123fa4e.slice/crio-f87f1c099410e073080d0b83b2af4c530bc6b4bd505343496f6141e115702092 WatchSource:0}: Error finding container f87f1c099410e073080d0b83b2af4c530bc6b4bd505343496f6141e115702092: Status 404 returned error can't find the container with id f87f1c099410e073080d0b83b2af4c530bc6b4bd505343496f6141e115702092 Oct 30 17:09:00.523765 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.523730 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:00.523841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.523799 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:00.523841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.523831 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:00.940794 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.940700 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" event={"ID":"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e","Type":"ContainerStarted","Data":"4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7"} Oct 30 17:09:00.940794 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:00.940752 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" event={"ID":"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e","Type":"ContainerStarted","Data":"f87f1c099410e073080d0b83b2af4c530bc6b4bd505343496f6141e115702092"} Oct 30 17:09:01.009819 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:01.009754 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" podStartSLOduration=2.009733833 podStartE2EDuration="2.009733833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:08:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:00.995134986 +0000 UTC m=+2850.145415989" watchObservedRunningTime="2026-10-30 17:09:01.009733833 +0000 UTC m=+2850.160014824" Oct 30 17:09:01.188835 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:01.188803 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:01.190245 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:01.190222 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:09:01.190372 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:01.190266 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:09:02.189372 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:02.189333 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:09:02.189829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:02.189391 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:09:03.189156 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:03.189113 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:09:03.189349 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:03.189172 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:09:04.188740 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:04.188704 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:09:04.189153 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:04.188762 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:09:05.189296 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:05.189263 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:09:05.189725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:05.189317 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:09:06.189082 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:06.189049 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:09:06.189299 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:06.189110 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:09:07.188483 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:07.188434 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 30 17:09:07.189042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:07.188524 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 30 17:09:08.194554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:08.194521 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:08.195081 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:08.195040 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:08.195696 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:08.195673 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:09:24.606391 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:09:24.606345 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:09:39.902654 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:39.902618 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx"] Oct 30 17:09:39.908802 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:39.908776 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:39.950485 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:39.950445 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx"] Oct 30 17:09:40.014743 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.014709 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e4048c-openshift-default-dockercfg-ck5hk\"" Oct 30 17:09:40.026455 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.026416 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.026825 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.026797 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c339b975-ee73-4069-898e-1589b161c6cc-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.027007 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.026965 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.027007 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.027004 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.027274 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.027037 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c339b975-ee73-4069-898e-1589b161c6cc-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.027274 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.027086 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.027274 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.027148 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.027274 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.027198 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4gqb\" (UniqueName: \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-kube-api-access-c4gqb\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.027652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.027340 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.128874 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.128831 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.128920 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c339b975-ee73-4069-898e-1589b161c6cc-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.128940 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.128969 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.128995 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c339b975-ee73-4069-898e-1589b161c6cc-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.129042 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129358 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.129098 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129358 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.129168 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4gqb\" (UniqueName: \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-kube-api-access-c4gqb\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129476 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.129372 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129540 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.129475 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-credential-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129613 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.129564 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-socket\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129684 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.129628 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-data\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129925 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.129888 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-certs\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.129925 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.129889 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c339b975-ee73-4069-898e-1589b161c6cc-istiod-ca-cert\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.131521 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.131502 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-envoy\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.131775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.131756 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c339b975-ee73-4069-898e-1589b161c6cc-istio-podinfo\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.152206 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.152175 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-istio-token\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.152824 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.152778 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4gqb\" (UniqueName: \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-kube-api-access-c4gqb\") pod \"e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.226022 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.225984 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:40.559305 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.559279 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx"] Oct 30 17:09:40.560034 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:09:40.560003 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc339b975_ee73_4069_898e_1589b161c6cc.slice/crio-15db4b557c021cd7033662ae859b7549f96afe0aead568321438da63dcb6e011 WatchSource:0}: Error finding container 15db4b557c021cd7033662ae859b7549f96afe0aead568321438da63dcb6e011: Status 404 returned error can't find the container with id 15db4b557c021cd7033662ae859b7549f96afe0aead568321438da63dcb6e011 Oct 30 17:09:40.563509 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.563468 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:40.563620 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.563545 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:40.563620 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:40.563581 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:09:41.299047 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:41.299013 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" event={"ID":"c339b975-ee73-4069-898e-1589b161c6cc","Type":"ContainerStarted","Data":"a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20"} Oct 30 17:09:41.299500 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:41.299075 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" event={"ID":"c339b975-ee73-4069-898e-1589b161c6cc","Type":"ContainerStarted","Data":"15db4b557c021cd7033662ae859b7549f96afe0aead568321438da63dcb6e011"} Oct 30 17:09:41.367880 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:41.367828 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" podStartSLOduration=2.367812702 podStartE2EDuration="2.367812702s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:09:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:09:41.353748827 +0000 UTC m=+2890.504029816" watchObservedRunningTime="2026-10-30 17:09:41.367812702 +0000 UTC m=+2890.518093689" Oct 30 17:09:42.228630 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:42.226983 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:42.233625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:42.231721 6400 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:09:42.233625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:42.231783 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" podUID="c339b975-ee73-4069-898e-1589b161c6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:09:43.234052 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:43.234012 6400 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:09:43.234705 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:43.234670 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" podUID="c339b975-ee73-4069-898e-1589b161c6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:09:43.890889 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:43.890853 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m"] Oct 30 17:09:44.237656 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:44.237112 6400 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:09:44.237656 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:44.237188 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" podUID="c339b975-ee73-4069-898e-1589b161c6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:09:45.235360 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:45.235322 6400 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:09:45.235697 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:45.235667 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" podUID="c339b975-ee73-4069-898e-1589b161c6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:09:46.227628 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:46.227036 6400 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" start-of-body= Oct 30 17:09:46.227628 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:46.227102 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" podUID="c339b975-ee73-4069-898e-1589b161c6cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.172:15021/healthz/ready\": dial tcp 10.134.0.172:15021: connect: connection refused" Oct 30 17:09:47.265198 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:47.265154 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:47.265718 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:47.265657 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:47.267194 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:47.267173 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:09:54.604098 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:09:54.604055 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:09:58.960784 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:58.960730 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" podUID="14738f12-f84d-4bf4-9c03-916c866acb95" containerName="main" containerID="cri-o://80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1" gracePeriod=45 Oct 30 17:09:59.266426 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.266394 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:09:59.309344 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.309311 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-model-cache\") pod \"14738f12-f84d-4bf4-9c03-916c866acb95\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " Oct 30 17:09:59.309536 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.309397 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/14738f12-f84d-4bf4-9c03-916c866acb95-tls-certs\" (UniqueName: \"kubernetes.io/secret/14738f12-f84d-4bf4-9c03-916c866acb95-tls-certs\") pod \"14738f12-f84d-4bf4-9c03-916c866acb95\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " Oct 30 17:09:59.309643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.309539 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-home\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-home\") pod \"14738f12-f84d-4bf4-9c03-916c866acb95\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " Oct 30 17:09:59.309643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.309577 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14738f12-f84d-4bf4-9c03-916c866acb95-kube-api-access-vk2vq\" (UniqueName: \"kubernetes.io/projected/14738f12-f84d-4bf4-9c03-916c866acb95-kube-api-access-vk2vq\") pod \"14738f12-f84d-4bf4-9c03-916c866acb95\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " Oct 30 17:09:59.309769 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.309644 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-tmp-dir\") pod \"14738f12-f84d-4bf4-9c03-916c866acb95\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " Oct 30 17:09:59.309769 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.309662 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-model-cache" pod "14738f12-f84d-4bf4-9c03-916c866acb95" (UID: "14738f12-f84d-4bf4-9c03-916c866acb95"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:09:59.309769 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.309735 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-dshm\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-dshm\") pod \"14738f12-f84d-4bf4-9c03-916c866acb95\" (UID: \"14738f12-f84d-4bf4-9c03-916c866acb95\") " Oct 30 17:09:59.309939 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.309859 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-home" pod "14738f12-f84d-4bf4-9c03-916c866acb95" (UID: "14738f12-f84d-4bf4-9c03-916c866acb95"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:09:59.310074 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.309989 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-tmp-dir" pod "14738f12-f84d-4bf4-9c03-916c866acb95" (UID: "14738f12-f84d-4bf4-9c03-916c866acb95"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:09:59.310377 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.310357 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:09:59.310459 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.310383 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:09:59.310459 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.310398 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:09:59.312140 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.312111 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-dshm" pod "14738f12-f84d-4bf4-9c03-916c866acb95" (UID: "14738f12-f84d-4bf4-9c03-916c866acb95"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:09:59.313001 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.312978 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/14738f12-f84d-4bf4-9c03-916c866acb95-tls-certs" pod "14738f12-f84d-4bf4-9c03-916c866acb95" (UID: "14738f12-f84d-4bf4-9c03-916c866acb95"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:09:59.313108 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.313097 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14738f12-f84d-4bf4-9c03-916c866acb95-kube-api-access-vk2vq" pod "14738f12-f84d-4bf4-9c03-916c866acb95" (UID: "14738f12-f84d-4bf4-9c03-916c866acb95"). InnerVolumeSpecName "kube-api-access-vk2vq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:09:59.412339 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.412307 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vk2vq\" (UniqueName: \"kubernetes.io/projected/14738f12-f84d-4bf4-9c03-916c866acb95-kube-api-access-vk2vq\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:09:59.412339 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.412340 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14738f12-f84d-4bf4-9c03-916c866acb95-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:09:59.412535 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.412357 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14738f12-f84d-4bf4-9c03-916c866acb95-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:09:59.487338 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.487307 6400 generic.go:356] "Generic (PLEG): container finished" podID="14738f12-f84d-4bf4-9c03-916c866acb95" containerID="80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1" exitCode=0 Oct 30 17:09:59.487532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.487350 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" Oct 30 17:09:59.487532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.487388 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" event={"ID":"14738f12-f84d-4bf4-9c03-916c866acb95","Type":"ContainerDied","Data":"80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1"} Oct 30 17:09:59.487532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.487432 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m" event={"ID":"14738f12-f84d-4bf4-9c03-916c866acb95","Type":"ContainerDied","Data":"68f8d666d0efa4ab0c853504d6b5c79b61583fb3afe29478a7c4f51ea5e32e19"} Oct 30 17:09:59.487532 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.487462 6400 scope.go:122] "RemoveContainer" containerID="80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1" Oct 30 17:09:59.500414 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.500393 6400 scope.go:122] "RemoveContainer" containerID="80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1" Oct 30 17:09:59.500749 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:09:59.500726 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1\": container with ID starting with 80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1 not found: ID does not exist" containerID="80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1" Oct 30 17:09:59.500807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.500760 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1"} err="failed to get container status \"80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1\": rpc error: code = NotFound desc = could not find container \"80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1\": container with ID starting with 80807b2a9ae5119e21fa34723922f3a47aca2b1fa7dd6224d315ffde94d46bf1 not found: ID does not exist" Oct 30 17:09:59.532611 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.532581 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m"] Oct 30 17:09:59.538044 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:09:59.538022 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-9e533f/sub-test-model-9e533f-kserve-587cffd96c-w446m"] Oct 30 17:10:00.968419 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:00.968388 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14738f12-f84d-4bf4-9c03-916c866acb95" path="/var/lib/kubelet/pods/14738f12-f84d-4bf4-9c03-916c866acb95/volumes" Oct 30 17:10:10.763394 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:10.763360 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz"] Oct 30 17:10:10.764022 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:10.763619 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" podUID="2ae4c919-e0af-4d39-98ef-5fb6eceab7d0" containerName="maas-api" containerID="cri-o://fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060" gracePeriod=30 Oct 30 17:10:11.142171 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.142141 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 17:10:11.287466 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.287418 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-maas-api-tls\") pod \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " Oct 30 17:10:11.287698 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.287548 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-openshift-service-ca\") pod \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " Oct 30 17:10:11.287779 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.287723 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-metrics-certs\") pod \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " Oct 30 17:10:11.287842 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.287782 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-kube-api-access-89gh4\" (UniqueName: \"kubernetes.io/projected/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-kube-api-access-89gh4\") pod \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\" (UID: \"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0\") " Oct 30 17:10:11.287966 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.287937 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-openshift-service-ca" pod "2ae4c919-e0af-4d39-98ef-5fb6eceab7d0" (UID: "2ae4c919-e0af-4d39-98ef-5fb6eceab7d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:11.288634 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.288590 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:11.290400 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.290370 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-metrics-certs" pod "2ae4c919-e0af-4d39-98ef-5fb6eceab7d0" (UID: "2ae4c919-e0af-4d39-98ef-5fb6eceab7d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:11.290586 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.290568 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-kube-api-access-89gh4" pod "2ae4c919-e0af-4d39-98ef-5fb6eceab7d0" (UID: "2ae4c919-e0af-4d39-98ef-5fb6eceab7d0"). InnerVolumeSpecName "kube-api-access-89gh4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:11.290798 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.290699 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-maas-api-tls" pod "2ae4c919-e0af-4d39-98ef-5fb6eceab7d0" (UID: "2ae4c919-e0af-4d39-98ef-5fb6eceab7d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:11.390530 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.390433 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:11.390530 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.390472 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:11.390530 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.390487 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89gh4\" (UniqueName: \"kubernetes.io/projected/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0-kube-api-access-89gh4\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:11.587876 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.587839 6400 generic.go:356] "Generic (PLEG): container finished" podID="2ae4c919-e0af-4d39-98ef-5fb6eceab7d0" containerID="fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060" exitCode=0 Oct 30 17:10:11.588069 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.587885 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" Oct 30 17:10:11.588069 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.587923 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" event={"ID":"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0","Type":"ContainerDied","Data":"fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060"} Oct 30 17:10:11.588069 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.587985 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz" event={"ID":"2ae4c919-e0af-4d39-98ef-5fb6eceab7d0","Type":"ContainerDied","Data":"af14a15ed8d699a329c9025624c40a553a988623230e691be5dcc79a91caeda9"} Oct 30 17:10:11.588069 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.588009 6400 scope.go:122] "RemoveContainer" containerID="fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060" Oct 30 17:10:11.600038 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.600015 6400 scope.go:122] "RemoveContainer" containerID="fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060" Oct 30 17:10:11.600475 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:10:11.600440 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060\": container with ID starting with fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060 not found: ID does not exist" containerID="fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060" Oct 30 17:10:11.600578 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.600477 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060"} err="failed to get container status \"fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060\": rpc error: code = NotFound desc = could not find container \"fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060\": container with ID starting with fc6263d3179880f6eded2057308066101afb8f37d9224adb13e9f93d0cbb9060 not found: ID does not exist" Oct 30 17:10:11.634082 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.634050 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz"] Oct 30 17:10:11.644060 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:11.643982 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-d693d0-74cfccfb8-5zhfz"] Oct 30 17:10:12.968327 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:12.968292 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ae4c919-e0af-4d39-98ef-5fb6eceab7d0" path="/var/lib/kubelet/pods/2ae4c919-e0af-4d39-98ef-5fb6eceab7d0/volumes" Oct 30 17:10:19.237474 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:19.237436 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x"] Oct 30 17:10:19.238137 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:19.238106 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" containerID="cri-o://e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d" gracePeriod=30 Oct 30 17:10:23.580308 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.580273 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66"] Oct 30 17:10:23.582100 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.582071 6400 cpu_manager.go:414] "containerMap: removing container" podUID="2ae4c919-e0af-4d39-98ef-5fb6eceab7d0" containerName="maas-api" Oct 30 17:10:23.582286 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.582273 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ae4c919-e0af-4d39-98ef-5fb6eceab7d0" containerName="maas-api" Oct 30 17:10:23.582440 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.582426 6400 cpu_manager.go:414] "containerMap: removing container" podUID="14738f12-f84d-4bf4-9c03-916c866acb95" containerName="main" Oct 30 17:10:23.582531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.582521 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14738f12-f84d-4bf4-9c03-916c866acb95" containerName="main" Oct 30 17:10:23.582898 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.582790 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="14738f12-f84d-4bf4-9c03-916c866acb95" containerName="main" Oct 30 17:10:23.583100 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.583086 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ae4c919-e0af-4d39-98ef-5fb6eceab7d0" containerName="maas-api" Oct 30 17:10:23.646816 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.646067 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66"] Oct 30 17:10:23.646816 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.646261 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:10:23.662527 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.662495 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-e4048c\"" Oct 30 17:10:23.672200 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.672172 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-e4048c-dockercfg-pmq77\"" Oct 30 17:10:23.715673 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.715624 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4"] Oct 30 17:10:23.720415 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.720391 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:10:23.742981 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.742945 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4"] Oct 30 17:10:23.783903 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.783803 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756535eb-8aac-4808-8a61-b2ac1b8813e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4\" (UID: \"756535eb-8aac-4808-8a61-b2ac1b8813e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:10:23.784313 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.784283 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f30a3fc-3120-441f-bb7b-717c3294e56c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66\" (UID: \"1f30a3fc-3120-441f-bb7b-717c3294e56c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:10:23.784452 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.784418 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-747tb\" (UniqueName: \"kubernetes.io/projected/1f30a3fc-3120-441f-bb7b-717c3294e56c-kube-api-access-747tb\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66\" (UID: \"1f30a3fc-3120-441f-bb7b-717c3294e56c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:10:23.784721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.784472 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlxb5\" (UniqueName: \"kubernetes.io/projected/756535eb-8aac-4808-8a61-b2ac1b8813e2-kube-api-access-nlxb5\") pod \"payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4\" (UID: \"756535eb-8aac-4808-8a61-b2ac1b8813e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:10:23.886418 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.886317 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-747tb\" (UniqueName: \"kubernetes.io/projected/1f30a3fc-3120-441f-bb7b-717c3294e56c-kube-api-access-747tb\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66\" (UID: \"1f30a3fc-3120-441f-bb7b-717c3294e56c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:10:23.886418 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.886377 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlxb5\" (UniqueName: \"kubernetes.io/projected/756535eb-8aac-4808-8a61-b2ac1b8813e2-kube-api-access-nlxb5\") pod \"payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4\" (UID: \"756535eb-8aac-4808-8a61-b2ac1b8813e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:10:23.886418 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.886419 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756535eb-8aac-4808-8a61-b2ac1b8813e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4\" (UID: \"756535eb-8aac-4808-8a61-b2ac1b8813e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:10:23.886813 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.886560 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f30a3fc-3120-441f-bb7b-717c3294e56c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66\" (UID: \"1f30a3fc-3120-441f-bb7b-717c3294e56c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:10:23.887268 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.887240 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756535eb-8aac-4808-8a61-b2ac1b8813e2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4\" (UID: \"756535eb-8aac-4808-8a61-b2ac1b8813e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:10:23.887380 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.887299 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f30a3fc-3120-441f-bb7b-717c3294e56c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66\" (UID: \"1f30a3fc-3120-441f-bb7b-717c3294e56c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:10:23.907543 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.907513 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlxb5\" (UniqueName: \"kubernetes.io/projected/756535eb-8aac-4808-8a61-b2ac1b8813e2-kube-api-access-nlxb5\") pod \"payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4\" (UID: \"756535eb-8aac-4808-8a61-b2ac1b8813e2\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:10:23.907853 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.907832 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-747tb\" (UniqueName: \"kubernetes.io/projected/1f30a3fc-3120-441f-bb7b-717c3294e56c-kube-api-access-747tb\") pod \"payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66\" (UID: \"1f30a3fc-3120-441f-bb7b-717c3294e56c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:10:23.970758 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:23.970730 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:10:24.040576 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.040540 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:10:24.212084 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.212006 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9"] Oct 30 17:10:24.218020 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.217995 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.236776 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.236747 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9"] Oct 30 17:10:24.240015 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.239990 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-698c71-serving-cert\"" Oct 30 17:10:24.281303 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.281271 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66"] Oct 30 17:10:24.293017 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.292985 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f2vxw\" (UniqueName: \"kubernetes.io/projected/40f37558-408c-4fea-a976-a3b2cb8bbe4d-kube-api-access-f2vxw\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.293196 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.293175 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.293473 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.293283 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f37558-408c-4fea-a976-a3b2cb8bbe4d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.293473 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.293375 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-metrics-certs\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.320567 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:10:24.320539 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f30a3fc_3120_441f_bb7b_717c3294e56c.slice/crio-254ac45fa3aa6e49dd800c00b50c1f11f3beee9dc963463045ad8eefa4171233 WatchSource:0}: Error finding container 254ac45fa3aa6e49dd800c00b50c1f11f3beee9dc963463045ad8eefa4171233: Status 404 returned error can't find the container with id 254ac45fa3aa6e49dd800c00b50c1f11f3beee9dc963463045ad8eefa4171233 Oct 30 17:10:24.379471 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.379436 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4"] Oct 30 17:10:24.380396 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:10:24.380360 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod756535eb_8aac_4808_8a61_b2ac1b8813e2.slice/crio-9850da1bfbd2683d18a25e78f81d4ff822c405b849df1c423d1aacfa838b982e WatchSource:0}: Error finding container 9850da1bfbd2683d18a25e78f81d4ff822c405b849df1c423d1aacfa838b982e: Status 404 returned error can't find the container with id 9850da1bfbd2683d18a25e78f81d4ff822c405b849df1c423d1aacfa838b982e Oct 30 17:10:24.396567 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.394739 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f2vxw\" (UniqueName: \"kubernetes.io/projected/40f37558-408c-4fea-a976-a3b2cb8bbe4d-kube-api-access-f2vxw\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.396567 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.394875 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.396567 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.394939 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f37558-408c-4fea-a976-a3b2cb8bbe4d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.396567 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.395029 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-metrics-certs\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.396567 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.396257 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f37558-408c-4fea-a976-a3b2cb8bbe4d-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.402549 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.402518 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-maas-api-tls\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.407124 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.407103 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-metrics-certs\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.417913 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.417869 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f2vxw\" (UniqueName: \"kubernetes.io/projected/40f37558-408c-4fea-a976-a3b2cb8bbe4d-kube-api-access-f2vxw\") pod \"maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.537222 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.537022 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:24.555109 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.554840 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 17:10:24.646866 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:10:24.646733 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:10:24.703390 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.703354 6400 generic.go:356] "Generic (PLEG): container finished" podID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerID="e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d" exitCode=0 Oct 30 17:10:24.703590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.703481 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" Oct 30 17:10:24.703590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.703506 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" event={"ID":"ca64f363-9b91-46d1-8e52-f35df4dc1a3b","Type":"ContainerDied","Data":"e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d"} Oct 30 17:10:24.703590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.703555 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x" event={"ID":"ca64f363-9b91-46d1-8e52-f35df4dc1a3b","Type":"ContainerDied","Data":"6683d7c3be57ba29c5d79a55bc0c1c2ec0b885db56f8dddb6d3d7e1c735b29f5"} Oct 30 17:10:24.703795 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.703594 6400 scope.go:122] "RemoveContainer" containerID="e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d" Oct 30 17:10:24.706085 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.706040 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" event={"ID":"756535eb-8aac-4808-8a61-b2ac1b8813e2","Type":"ContainerStarted","Data":"dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be"} Oct 30 17:10:24.706221 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.706099 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" event={"ID":"756535eb-8aac-4808-8a61-b2ac1b8813e2","Type":"ContainerStarted","Data":"9850da1bfbd2683d18a25e78f81d4ff822c405b849df1c423d1aacfa838b982e"} Oct 30 17:10:24.706460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.706438 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:10:24.708489 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.708461 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" event={"ID":"1f30a3fc-3120-441f-bb7b-717c3294e56c","Type":"ContainerStarted","Data":"e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4"} Oct 30 17:10:24.708652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.708513 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" event={"ID":"1f30a3fc-3120-441f-bb7b-717c3294e56c","Type":"ContainerStarted","Data":"254ac45fa3aa6e49dd800c00b50c1f11f3beee9dc963463045ad8eefa4171233"} Oct 30 17:10:24.709362 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.709336 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:10:24.709892 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.709866 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-podinfo\") pod \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " Oct 30 17:10:24.709996 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.709921 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-token\" (UniqueName: \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-token\") pod \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " Oct 30 17:10:24.709996 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.709961 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-socket\") pod \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " Oct 30 17:10:24.710120 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.709997 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-envoy\") pod \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " Oct 30 17:10:24.710120 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.710059 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istiod-ca-cert\") pod \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " Oct 30 17:10:24.710237 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.710144 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-certs\") pod \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " Oct 30 17:10:24.710290 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.710257 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-kube-api-access-n7q76\" (UniqueName: \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-kube-api-access-n7q76\") pod \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " Oct 30 17:10:24.710348 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.710320 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-credential-socket\") pod \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " Oct 30 17:10:24.710406 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.710391 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-data\") pod \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\" (UID: \"ca64f363-9b91-46d1-8e52-f35df4dc1a3b\") " Oct 30 17:10:24.712180 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.711984 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-socket" pod "ca64f363-9b91-46d1-8e52-f35df4dc1a3b" (UID: "ca64f363-9b91-46d1-8e52-f35df4dc1a3b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:24.715026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.712224 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-certs" pod "ca64f363-9b91-46d1-8e52-f35df4dc1a3b" (UID: "ca64f363-9b91-46d1-8e52-f35df4dc1a3b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:24.715026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.712270 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istiod-ca-cert" pod "ca64f363-9b91-46d1-8e52-f35df4dc1a3b" (UID: "ca64f363-9b91-46d1-8e52-f35df4dc1a3b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:24.715026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.712420 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-credential-socket" pod "ca64f363-9b91-46d1-8e52-f35df4dc1a3b" (UID: "ca64f363-9b91-46d1-8e52-f35df4dc1a3b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:24.715026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.712680 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-data" pod "ca64f363-9b91-46d1-8e52-f35df4dc1a3b" (UID: "ca64f363-9b91-46d1-8e52-f35df4dc1a3b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:24.715026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.712968 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.715026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.712990 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.715026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.713004 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.715026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.713016 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.715026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.713029 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.722012 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.721741 6400 scope.go:122] "RemoveContainer" containerID="e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d" Oct 30 17:10:24.722328 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:10:24.722228 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d\": container with ID starting with e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d not found: ID does not exist" containerID="e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d" Oct 30 17:10:24.722328 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.722264 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d"} err="failed to get container status \"e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d\": rpc error: code = NotFound desc = could not find container \"e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d\": container with ID starting with e7e19510bd2f816aa5b02d117a93f3ea6123d3cc2c2c54daea489b5bc981273d not found: ID does not exist" Oct 30 17:10:24.723129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.723104 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-envoy" pod "ca64f363-9b91-46d1-8e52-f35df4dc1a3b" (UID: "ca64f363-9b91-46d1-8e52-f35df4dc1a3b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:10:24.723836 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.723801 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-token" pod "ca64f363-9b91-46d1-8e52-f35df4dc1a3b" (UID: "ca64f363-9b91-46d1-8e52-f35df4dc1a3b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:24.724128 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.724092 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-kube-api-access-n7q76" pod "ca64f363-9b91-46d1-8e52-f35df4dc1a3b" (UID: "ca64f363-9b91-46d1-8e52-f35df4dc1a3b"). InnerVolumeSpecName "kube-api-access-n7q76". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:24.725014 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.724992 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-podinfo" pod "ca64f363-9b91-46d1-8e52-f35df4dc1a3b" (UID: "ca64f363-9b91-46d1-8e52-f35df4dc1a3b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:10:24.756768 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.756706 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" podStartSLOduration=1.756693742 podStartE2EDuration="1.756693742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:24.75495054 +0000 UTC m=+2933.905231529" watchObservedRunningTime="2026-10-30 17:10:24.756693742 +0000 UTC m=+2933.906974731" Oct 30 17:10:24.793416 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.793358 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" podStartSLOduration=1.793342189 podStartE2EDuration="1.793342189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:24.790275343 +0000 UTC m=+2933.940556335" watchObservedRunningTime="2026-10-30 17:10:24.793342189 +0000 UTC m=+2933.943623178" Oct 30 17:10:24.815277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.815246 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.815277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.815271 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.815277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.815281 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.815563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.815295 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7q76\" (UniqueName: \"kubernetes.io/projected/ca64f363-9b91-46d1-8e52-f35df4dc1a3b-kube-api-access-n7q76\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:24.876766 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:24.876675 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9"] Oct 30 17:10:24.876913 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:10:24.876780 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40f37558_408c_4fea_a976_a3b2cb8bbe4d.slice/crio-d2226f12061818164ca48841ae435b8ba5f7ab84a26317bd0fcea9ca42857e89 WatchSource:0}: Error finding container d2226f12061818164ca48841ae435b8ba5f7ab84a26317bd0fcea9ca42857e89: Status 404 returned error can't find the container with id d2226f12061818164ca48841ae435b8ba5f7ab84a26317bd0fcea9ca42857e89 Oct 30 17:10:25.043446 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:25.043408 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x"] Oct 30 17:10:25.048844 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:25.048818 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-d693d0-openshift-default-cb8d589c7-fl67x"] Oct 30 17:10:25.716361 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:25.716324 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" event={"ID":"40f37558-408c-4fea-a976-a3b2cb8bbe4d","Type":"ContainerStarted","Data":"d2226f12061818164ca48841ae435b8ba5f7ab84a26317bd0fcea9ca42857e89"} Oct 30 17:10:26.740683 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:26.740637 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" event={"ID":"40f37558-408c-4fea-a976-a3b2cb8bbe4d","Type":"ContainerStarted","Data":"a217cdca64d704fc3a84cc8ded18fa971f043ebeaa72e5f52552266e3e480152"} Oct 30 17:10:26.741231 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:26.740890 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:26.782220 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:26.782171 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" podStartSLOduration=1.876074448 podStartE2EDuration="2.782154472s" totalImagesPullingTime="906.080024ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:26.777978415 +0000 UTC m=+2935.928259405" watchObservedRunningTime="2026-10-30 17:10:26.782154472 +0000 UTC m=+2935.932435462" Oct 30 17:10:26.969387 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:26.969347 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" path="/var/lib/kubelet/pods/ca64f363-9b91-46d1-8e52-f35df4dc1a3b/volumes" Oct 30 17:10:32.755118 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:32.755089 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:10:35.719643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:35.719612 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:10:36.744179 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:36.744148 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:10:43.803718 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:43.803679 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2"] Oct 30 17:10:43.804348 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:43.804334 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" Oct 30 17:10:43.804393 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:43.804349 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" Oct 30 17:10:43.804481 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:43.804472 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca64f363-9b91-46d1-8e52-f35df4dc1a3b" containerName="istio-proxy" Oct 30 17:10:43.809617 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:43.809582 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:10:43.841375 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:43.841345 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4ac125\"" Oct 30 17:10:43.848758 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:43.848729 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4ac125-dockercfg-d2qlt\"" Oct 30 17:10:43.854082 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:43.854055 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2"] Oct 30 17:10:43.912508 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:43.912466 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2\" (UID: \"ddc37d3c-fb6d-4b17-94c3-1619c26349c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:10:43.912801 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:43.912666 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b495q\" (UniqueName: \"kubernetes.io/projected/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-kube-api-access-b495q\") pod \"payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2\" (UID: \"ddc37d3c-fb6d-4b17-94c3-1619c26349c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:10:44.014556 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:44.014521 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2\" (UID: \"ddc37d3c-fb6d-4b17-94c3-1619c26349c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:10:44.014798 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:44.014641 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b495q\" (UniqueName: \"kubernetes.io/projected/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-kube-api-access-b495q\") pod \"payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2\" (UID: \"ddc37d3c-fb6d-4b17-94c3-1619c26349c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:10:44.015336 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:44.015312 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2\" (UID: \"ddc37d3c-fb6d-4b17-94c3-1619c26349c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:10:44.038567 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:44.038541 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b495q\" (UniqueName: \"kubernetes.io/projected/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-kube-api-access-b495q\") pod \"payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2\" (UID: \"ddc37d3c-fb6d-4b17-94c3-1619c26349c5\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:10:44.124697 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:44.124593 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:10:44.456644 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:44.456594 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2"] Oct 30 17:10:44.892015 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:44.891960 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" event={"ID":"ddc37d3c-fb6d-4b17-94c3-1619c26349c5","Type":"ContainerStarted","Data":"e6e03e333c76c73ff16be90e57ebca7a740251831a178331f6abcedd6e4cc931"} Oct 30 17:10:44.892450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:44.892026 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" event={"ID":"ddc37d3c-fb6d-4b17-94c3-1619c26349c5","Type":"ContainerStarted","Data":"e78019e4d76436eeccabe326bc97794cfda7319ec21bc54974c9474d23207774"} Oct 30 17:10:44.892450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:44.892186 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:10:44.937228 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:44.937174 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" podStartSLOduration=1.9371609680000001 podStartE2EDuration="1.937160968s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:44.932100944 +0000 UTC m=+2954.082381932" watchObservedRunningTime="2026-10-30 17:10:44.937160968 +0000 UTC m=+2954.087441955" Oct 30 17:10:48.359509 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.359467 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh"] Oct 30 17:10:48.364047 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.364028 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.386203 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.386176 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-de18d5-serving-cert\"" Oct 30 17:10:48.398323 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.398294 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh"] Oct 30 17:10:48.486932 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.486898 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.487135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.486969 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.487135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.487121 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-metrics-certs\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.487230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.487186 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsrm7\" (UniqueName: \"kubernetes.io/projected/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-kube-api-access-vsrm7\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.589484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.588727 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.589484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.588816 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.589484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.588894 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-metrics-certs\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.589484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.589065 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsrm7\" (UniqueName: \"kubernetes.io/projected/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-kube-api-access-vsrm7\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.590521 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.590498 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.604634 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.597652 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-metrics-certs\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.604634 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.598281 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-maas-api-tls\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.621153 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.621059 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsrm7\" (UniqueName: \"kubernetes.io/projected/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-kube-api-access-vsrm7\") pod \"maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.679020 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.678986 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:48.988209 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:48.988184 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh"] Oct 30 17:10:48.988894 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:10:48.988869 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e2923a4_b7a1_49b9_95b1_1ff146f5f39d.slice/crio-02cd90835b05b0cf8e8f2f61a55d26cb102212b8e9dde64464f9e35c214407ab WatchSource:0}: Error finding container 02cd90835b05b0cf8e8f2f61a55d26cb102212b8e9dde64464f9e35c214407ab: Status 404 returned error can't find the container with id 02cd90835b05b0cf8e8f2f61a55d26cb102212b8e9dde64464f9e35c214407ab Oct 30 17:10:49.934667 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:49.934587 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" event={"ID":"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d","Type":"ContainerStarted","Data":"cc14a57ffb5ca27bded6a6882872eea299e5b1bfca18cf57681e666bbdde2ffa"} Oct 30 17:10:49.935106 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:49.934698 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:49.935106 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:49.934721 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" event={"ID":"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d","Type":"ContainerStarted","Data":"02cd90835b05b0cf8e8f2f61a55d26cb102212b8e9dde64464f9e35c214407ab"} Oct 30 17:10:49.976447 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:49.976392 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" podStartSLOduration=1.579539311 podStartE2EDuration="1.976377292s" totalImagesPullingTime="396.837981ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:10:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:10:49.974883388 +0000 UTC m=+2959.125164377" watchObservedRunningTime="2026-10-30 17:10:49.976377292 +0000 UTC m=+2959.126658279" Oct 30 17:10:54.600650 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:10:54.600597 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:10:55.901581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:55.901553 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:10:55.947465 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:55.947435 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:10:57.496835 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:57.496802 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk"] Oct 30 17:10:57.497334 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:57.497099 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" podUID="7a348964-7aa9-4ca7-98b2-62992f766d4b" containerName="maas-api" containerID="cri-o://855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70" gracePeriod=30 Oct 30 17:10:57.660878 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:57.660831 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw"] Oct 30 17:10:57.661145 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:57.661094 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" podUID="f12e857f-ab74-4122-abe5-5c0297070fc0" containerName="payload-processing" containerID="cri-o://b8f442d3198ce32f9433ede0b554afee649f63a2deb5b5490ffc2dc6a4243da7" gracePeriod=30 Oct 30 17:10:57.687620 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:57.687577 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24"] Oct 30 17:10:57.688078 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:57.688027 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" podUID="aa869356-749f-4c10-b350-f1aa6fc87baf" containerName="payload-pre-processing" containerID="cri-o://33ac69cf10e7eaa4f822a7c6809c174b289923321b1ed0aca954bfd89d402f39" gracePeriod=30 Oct 30 17:10:57.921175 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:57.920777 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:10:58.036837 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.035231 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7a348964-7aa9-4ca7-98b2-62992f766d4b-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a348964-7aa9-4ca7-98b2-62992f766d4b-openshift-service-ca\") pod \"7a348964-7aa9-4ca7-98b2-62992f766d4b\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " Oct 30 17:10:58.036837 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.035319 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-maas-api-tls\") pod \"7a348964-7aa9-4ca7-98b2-62992f766d4b\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " Oct 30 17:10:58.036837 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.035398 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7a348964-7aa9-4ca7-98b2-62992f766d4b-kube-api-access-9dfl8\" (UniqueName: \"kubernetes.io/projected/7a348964-7aa9-4ca7-98b2-62992f766d4b-kube-api-access-9dfl8\") pod \"7a348964-7aa9-4ca7-98b2-62992f766d4b\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " Oct 30 17:10:58.036837 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.035666 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-metrics-certs\") pod \"7a348964-7aa9-4ca7-98b2-62992f766d4b\" (UID: \"7a348964-7aa9-4ca7-98b2-62992f766d4b\") " Oct 30 17:10:58.057509 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.057173 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a348964-7aa9-4ca7-98b2-62992f766d4b-openshift-service-ca" pod "7a348964-7aa9-4ca7-98b2-62992f766d4b" (UID: "7a348964-7aa9-4ca7-98b2-62992f766d4b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:58.099404 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.095742 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-maas-api-tls" pod "7a348964-7aa9-4ca7-98b2-62992f766d4b" (UID: "7a348964-7aa9-4ca7-98b2-62992f766d4b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:58.110111 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.110055 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-metrics-certs" pod "7a348964-7aa9-4ca7-98b2-62992f766d4b" (UID: "7a348964-7aa9-4ca7-98b2-62992f766d4b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:10:58.112306 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.111955 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a348964-7aa9-4ca7-98b2-62992f766d4b-kube-api-access-9dfl8" pod "7a348964-7aa9-4ca7-98b2-62992f766d4b" (UID: "7a348964-7aa9-4ca7-98b2-62992f766d4b"). InnerVolumeSpecName "kube-api-access-9dfl8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:58.140625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.137434 6400 generic.go:356] "Generic (PLEG): container finished" podID="7a348964-7aa9-4ca7-98b2-62992f766d4b" containerID="855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70" exitCode=0 Oct 30 17:10:58.140625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.137624 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" event={"ID":"7a348964-7aa9-4ca7-98b2-62992f766d4b","Type":"ContainerDied","Data":"855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70"} Oct 30 17:10:58.140625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.137683 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" event={"ID":"7a348964-7aa9-4ca7-98b2-62992f766d4b","Type":"ContainerDied","Data":"c1c144b2235e20ce10506bee371cad94b201063f5bd399d63ab0c58adcabe140"} Oct 30 17:10:58.140625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.137715 6400 scope.go:122] "RemoveContainer" containerID="855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70" Oct 30 17:10:58.140625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.137896 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk" Oct 30 17:10:58.150158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.149877 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a348964-7aa9-4ca7-98b2-62992f766d4b-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:58.150158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.149907 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:58.150158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.149924 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dfl8\" (UniqueName: \"kubernetes.io/projected/7a348964-7aa9-4ca7-98b2-62992f766d4b-kube-api-access-9dfl8\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:58.150158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.149941 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a348964-7aa9-4ca7-98b2-62992f766d4b-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:58.152118 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.152078 6400 generic.go:356] "Generic (PLEG): container finished" podID="aa869356-749f-4c10-b350-f1aa6fc87baf" containerID="33ac69cf10e7eaa4f822a7c6809c174b289923321b1ed0aca954bfd89d402f39" exitCode=0 Oct 30 17:10:58.152206 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.152186 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" event={"ID":"aa869356-749f-4c10-b350-f1aa6fc87baf","Type":"ContainerDied","Data":"33ac69cf10e7eaa4f822a7c6809c174b289923321b1ed0aca954bfd89d402f39"} Oct 30 17:10:58.154362 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.154340 6400 generic.go:356] "Generic (PLEG): container finished" podID="f12e857f-ab74-4122-abe5-5c0297070fc0" containerID="b8f442d3198ce32f9433ede0b554afee649f63a2deb5b5490ffc2dc6a4243da7" exitCode=0 Oct 30 17:10:58.154548 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.154397 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" event={"ID":"f12e857f-ab74-4122-abe5-5c0297070fc0","Type":"ContainerDied","Data":"b8f442d3198ce32f9433ede0b554afee649f63a2deb5b5490ffc2dc6a4243da7"} Oct 30 17:10:58.164575 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.164544 6400 scope.go:122] "RemoveContainer" containerID="855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70" Oct 30 17:10:58.166968 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.166950 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:10:58.167909 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:10:58.167884 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70\": container with ID starting with 855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70 not found: ID does not exist" containerID="855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70" Oct 30 17:10:58.168015 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.167916 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70"} err="failed to get container status \"855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70\": rpc error: code = NotFound desc = could not find container \"855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70\": container with ID starting with 855a83fc62a87d1eec0970d1f69232d4341905f754652edaab5f037f11b68b70 not found: ID does not exist" Oct 30 17:10:58.190389 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.190361 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:10:58.201481 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.201443 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk"] Oct 30 17:10:58.206032 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.206013 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-b16279-747f9bf64d-cvgdk"] Oct 30 17:10:58.251446 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.251414 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa869356-749f-4c10-b350-f1aa6fc87baf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa869356-749f-4c10-b350-f1aa6fc87baf-plugins-config-volume\") pod \"aa869356-749f-4c10-b350-f1aa6fc87baf\" (UID: \"aa869356-749f-4c10-b350-f1aa6fc87baf\") " Oct 30 17:10:58.251733 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.251690 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f12e857f-ab74-4122-abe5-5c0297070fc0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12e857f-ab74-4122-abe5-5c0297070fc0-plugins-config-volume\") pod \"f12e857f-ab74-4122-abe5-5c0297070fc0\" (UID: \"f12e857f-ab74-4122-abe5-5c0297070fc0\") " Oct 30 17:10:58.251866 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.251795 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa869356-749f-4c10-b350-f1aa6fc87baf-plugins-config-volume" pod "aa869356-749f-4c10-b350-f1aa6fc87baf" (UID: "aa869356-749f-4c10-b350-f1aa6fc87baf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:58.251866 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.251836 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f12e857f-ab74-4122-abe5-5c0297070fc0-kube-api-access-2scxr\" (UniqueName: \"kubernetes.io/projected/f12e857f-ab74-4122-abe5-5c0297070fc0-kube-api-access-2scxr\") pod \"f12e857f-ab74-4122-abe5-5c0297070fc0\" (UID: \"f12e857f-ab74-4122-abe5-5c0297070fc0\") " Oct 30 17:10:58.252005 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.251990 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa869356-749f-4c10-b350-f1aa6fc87baf-kube-api-access-bt27c\" (UniqueName: \"kubernetes.io/projected/aa869356-749f-4c10-b350-f1aa6fc87baf-kube-api-access-bt27c\") pod \"aa869356-749f-4c10-b350-f1aa6fc87baf\" (UID: \"aa869356-749f-4c10-b350-f1aa6fc87baf\") " Oct 30 17:10:58.252126 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.252102 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f12e857f-ab74-4122-abe5-5c0297070fc0-plugins-config-volume" pod "f12e857f-ab74-4122-abe5-5c0297070fc0" (UID: "f12e857f-ab74-4122-abe5-5c0297070fc0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:10:58.253325 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.253086 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa869356-749f-4c10-b350-f1aa6fc87baf-plugins-config-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:58.253325 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.253112 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f12e857f-ab74-4122-abe5-5c0297070fc0-plugins-config-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:58.254347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.254322 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f12e857f-ab74-4122-abe5-5c0297070fc0-kube-api-access-2scxr" pod "f12e857f-ab74-4122-abe5-5c0297070fc0" (UID: "f12e857f-ab74-4122-abe5-5c0297070fc0"). InnerVolumeSpecName "kube-api-access-2scxr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:58.254454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.254430 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa869356-749f-4c10-b350-f1aa6fc87baf-kube-api-access-bt27c" pod "aa869356-749f-4c10-b350-f1aa6fc87baf" (UID: "aa869356-749f-4c10-b350-f1aa6fc87baf"). InnerVolumeSpecName "kube-api-access-bt27c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:10:58.354782 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.354700 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2scxr\" (UniqueName: \"kubernetes.io/projected/f12e857f-ab74-4122-abe5-5c0297070fc0-kube-api-access-2scxr\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:58.354782 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.354726 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bt27c\" (UniqueName: \"kubernetes.io/projected/aa869356-749f-4c10-b350-f1aa6fc87baf-kube-api-access-bt27c\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:10:58.968105 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:58.968068 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7a348964-7aa9-4ca7-98b2-62992f766d4b" path="/var/lib/kubelet/pods/7a348964-7aa9-4ca7-98b2-62992f766d4b/volumes" Oct 30 17:10:59.164962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:59.164931 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" Oct 30 17:10:59.164962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:59.164945 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24" event={"ID":"aa869356-749f-4c10-b350-f1aa6fc87baf","Type":"ContainerDied","Data":"6e40c8e4fc97f3f890becae73fb525fe6ec366e8154c192d7ce532ac3e6ff756"} Oct 30 17:10:59.165216 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:59.165009 6400 scope.go:122] "RemoveContainer" containerID="33ac69cf10e7eaa4f822a7c6809c174b289923321b1ed0aca954bfd89d402f39" Oct 30 17:10:59.166525 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:59.166501 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" event={"ID":"f12e857f-ab74-4122-abe5-5c0297070fc0","Type":"ContainerDied","Data":"9773671e55bfc59632caff8075c981bee9ded77ee7acf83d4c08d1e82fe248b1"} Oct 30 17:10:59.166713 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:59.166511 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw" Oct 30 17:10:59.176813 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:59.176793 6400 scope.go:122] "RemoveContainer" containerID="b8f442d3198ce32f9433ede0b554afee649f63a2deb5b5490ffc2dc6a4243da7" Oct 30 17:10:59.216681 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:59.216654 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24"] Oct 30 17:10:59.227671 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:59.227611 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-b16279-868cd65f55-hdc24"] Oct 30 17:10:59.253112 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:59.252926 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw"] Oct 30 17:10:59.256728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:10:59.256700 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-b16279-85f8bf6964-b2gpw"] Oct 30 17:11:00.972442 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:00.971339 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa869356-749f-4c10-b350-f1aa6fc87baf" path="/var/lib/kubelet/pods/aa869356-749f-4c10-b350-f1aa6fc87baf/volumes" Oct 30 17:11:00.972442 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:00.972075 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f12e857f-ab74-4122-abe5-5c0297070fc0" path="/var/lib/kubelet/pods/f12e857f-ab74-4122-abe5-5c0297070fc0/volumes" Oct 30 17:11:07.890772 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:07.890724 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk"] Oct 30 17:11:07.891211 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:07.891001 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" containerID="cri-o://efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722" gracePeriod=30 Oct 30 17:11:13.179446 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.179384 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 17:11:13.282468 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.282432 6400 generic.go:356] "Generic (PLEG): container finished" podID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerID="efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722" exitCode=0 Oct 30 17:11:13.282721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.282514 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" Oct 30 17:11:13.282721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.282517 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" event={"ID":"7fc3c304-52a0-4f96-8f6b-021c866e3f0a","Type":"ContainerDied","Data":"efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722"} Oct 30 17:11:13.282721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.282666 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk" event={"ID":"7fc3c304-52a0-4f96-8f6b-021c866e3f0a","Type":"ContainerDied","Data":"e52fa04782320c93ee8f3bc1b5d5f2e03da912523cfcc134a3d4d452bc7e5d0f"} Oct 30 17:11:13.282721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.282704 6400 scope.go:122] "RemoveContainer" containerID="efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722" Oct 30 17:11:13.287345 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.287324 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-podinfo\") pod \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " Oct 30 17:11:13.287468 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.287386 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-credential-socket\") pod \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " Oct 30 17:11:13.287630 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.287596 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-token\" (UniqueName: \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-token\") pod \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " Oct 30 17:11:13.287698 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.287674 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-data\") pod \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " Oct 30 17:11:13.287753 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.287729 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-credential-socket" pod "7fc3c304-52a0-4f96-8f6b-021c866e3f0a" (UID: "7fc3c304-52a0-4f96-8f6b-021c866e3f0a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:13.287798 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.287784 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-envoy\") pod \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " Oct 30 17:11:13.287850 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.287822 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-kube-api-access-gvb6k\" (UniqueName: \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-kube-api-access-gvb6k\") pod \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " Oct 30 17:11:13.287912 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.287895 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istiod-ca-cert\") pod \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " Oct 30 17:11:13.287973 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.287935 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-certs\") pod \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " Oct 30 17:11:13.288032 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.287974 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-socket\") pod \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\" (UID: \"7fc3c304-52a0-4f96-8f6b-021c866e3f0a\") " Oct 30 17:11:13.288085 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.288062 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-data" pod "7fc3c304-52a0-4f96-8f6b-021c866e3f0a" (UID: "7fc3c304-52a0-4f96-8f6b-021c866e3f0a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:13.288273 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.288248 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-certs" pod "7fc3c304-52a0-4f96-8f6b-021c866e3f0a" (UID: "7fc3c304-52a0-4f96-8f6b-021c866e3f0a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:13.288467 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.288449 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-socket" pod "7fc3c304-52a0-4f96-8f6b-021c866e3f0a" (UID: "7fc3c304-52a0-4f96-8f6b-021c866e3f0a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:13.288901 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.288876 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istiod-ca-cert" pod "7fc3c304-52a0-4f96-8f6b-021c866e3f0a" (UID: "7fc3c304-52a0-4f96-8f6b-021c866e3f0a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:13.288901 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.288890 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:13.289205 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.288916 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:13.289205 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.288931 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:13.289205 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.288945 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:13.290335 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.290311 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-envoy" pod "7fc3c304-52a0-4f96-8f6b-021c866e3f0a" (UID: "7fc3c304-52a0-4f96-8f6b-021c866e3f0a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:11:13.290335 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.290330 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-token" pod "7fc3c304-52a0-4f96-8f6b-021c866e3f0a" (UID: "7fc3c304-52a0-4f96-8f6b-021c866e3f0a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:13.290630 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.290580 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-kube-api-access-gvb6k" pod "7fc3c304-52a0-4f96-8f6b-021c866e3f0a" (UID: "7fc3c304-52a0-4f96-8f6b-021c866e3f0a"). InnerVolumeSpecName "kube-api-access-gvb6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:13.290965 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.290950 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-podinfo" pod "7fc3c304-52a0-4f96-8f6b-021c866e3f0a" (UID: "7fc3c304-52a0-4f96-8f6b-021c866e3f0a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:11:13.294734 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.294713 6400 scope.go:122] "RemoveContainer" containerID="efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722" Oct 30 17:11:13.295058 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:11:13.295040 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722\": container with ID starting with efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722 not found: ID does not exist" containerID="efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722" Oct 30 17:11:13.295129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.295068 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722"} err="failed to get container status \"efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722\": rpc error: code = NotFound desc = could not find container \"efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722\": container with ID starting with efa4cdd3a87f7b17247a8f9d9db02c74645326086c5c8e8559ab0370b11d8722 not found: ID does not exist" Oct 30 17:11:13.390754 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.390700 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:13.390754 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.390749 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:13.390754 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.390759 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gvb6k\" (UniqueName: \"kubernetes.io/projected/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-kube-api-access-gvb6k\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:13.390754 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.390771 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:13.391060 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.390780 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7fc3c304-52a0-4f96-8f6b-021c866e3f0a-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:13.630803 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.630774 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk"] Oct 30 17:11:13.640650 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:13.640621 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-gjgfk"] Oct 30 17:11:14.971433 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:14.971398 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" path="/var/lib/kubelet/pods/7fc3c304-52a0-4f96-8f6b-021c866e3f0a/volumes" Oct 30 17:11:23.717347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:23.717307 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd"] Oct 30 17:11:23.717911 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:23.717561 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" podUID="a522dd46-312d-47ac-8a11-07a76546094a" containerName="maas-api" containerID="cri-o://19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294" gracePeriod=30 Oct 30 17:11:24.047123 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.047094 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:11:24.160384 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.160356 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-maas-api-tls\") pod \"a522dd46-312d-47ac-8a11-07a76546094a\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " Oct 30 17:11:24.160579 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.160407 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-metrics-certs\") pod \"a522dd46-312d-47ac-8a11-07a76546094a\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " Oct 30 17:11:24.160579 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.160564 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a522dd46-312d-47ac-8a11-07a76546094a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a522dd46-312d-47ac-8a11-07a76546094a-openshift-service-ca\") pod \"a522dd46-312d-47ac-8a11-07a76546094a\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " Oct 30 17:11:24.160712 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.160653 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a522dd46-312d-47ac-8a11-07a76546094a-kube-api-access-vj2wh\" (UniqueName: \"kubernetes.io/projected/a522dd46-312d-47ac-8a11-07a76546094a-kube-api-access-vj2wh\") pod \"a522dd46-312d-47ac-8a11-07a76546094a\" (UID: \"a522dd46-312d-47ac-8a11-07a76546094a\") " Oct 30 17:11:24.160993 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.160965 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a522dd46-312d-47ac-8a11-07a76546094a-openshift-service-ca" pod "a522dd46-312d-47ac-8a11-07a76546094a" (UID: "a522dd46-312d-47ac-8a11-07a76546094a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:11:24.161329 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.161310 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a522dd46-312d-47ac-8a11-07a76546094a-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:24.162978 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.162956 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a522dd46-312d-47ac-8a11-07a76546094a-kube-api-access-vj2wh" pod "a522dd46-312d-47ac-8a11-07a76546094a" (UID: "a522dd46-312d-47ac-8a11-07a76546094a"). InnerVolumeSpecName "kube-api-access-vj2wh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:11:24.163271 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.163247 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-metrics-certs" pod "a522dd46-312d-47ac-8a11-07a76546094a" (UID: "a522dd46-312d-47ac-8a11-07a76546094a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:11:24.163332 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.163265 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-maas-api-tls" pod "a522dd46-312d-47ac-8a11-07a76546094a" (UID: "a522dd46-312d-47ac-8a11-07a76546094a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:11:24.279683 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.278805 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:24.279683 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.278840 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a522dd46-312d-47ac-8a11-07a76546094a-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:24.279683 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.278854 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj2wh\" (UniqueName: \"kubernetes.io/projected/a522dd46-312d-47ac-8a11-07a76546094a-kube-api-access-vj2wh\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:11:24.374624 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.373499 6400 generic.go:356] "Generic (PLEG): container finished" podID="a522dd46-312d-47ac-8a11-07a76546094a" containerID="19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294" exitCode=0 Oct 30 17:11:24.374624 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.373551 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" event={"ID":"a522dd46-312d-47ac-8a11-07a76546094a","Type":"ContainerDied","Data":"19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294"} Oct 30 17:11:24.374624 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.373651 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" event={"ID":"a522dd46-312d-47ac-8a11-07a76546094a","Type":"ContainerDied","Data":"4c31e27753902992d3da3e478884cae08932d1ead966ef23754da3f4fc73a13e"} Oct 30 17:11:24.374624 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.373691 6400 scope.go:122] "RemoveContainer" containerID="19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294" Oct 30 17:11:24.374624 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.373876 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd" Oct 30 17:11:24.389782 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.389754 6400 scope.go:122] "RemoveContainer" containerID="19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294" Oct 30 17:11:24.390080 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:11:24.390060 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294\": container with ID starting with 19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294 not found: ID does not exist" containerID="19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294" Oct 30 17:11:24.390167 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.390089 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294"} err="failed to get container status \"19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294\": rpc error: code = NotFound desc = could not find container \"19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294\": container with ID starting with 19c3ffc295dca17b52146d925ba2212318cdf5f6d336e1801d243b2ff2715294 not found: ID does not exist" Oct 30 17:11:24.434286 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.434249 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd"] Oct 30 17:11:24.447891 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:24.447858 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-e1e830-8679d9dcf4-rpdhd"] Oct 30 17:11:24.605229 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:11:24.605148 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:11:25.013229 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:25.013191 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a522dd46-312d-47ac-8a11-07a76546094a" path="/var/lib/kubelet/pods/a522dd46-312d-47ac-8a11-07a76546094a/volumes" Oct 30 17:11:32.305341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:32.305308 6400 scope.go:122] "RemoveContainer" containerID="162c9ac6afbeba031a7b12bfef141daede487cbec0f0d9bd8f46ce71f48aa132" Oct 30 17:11:36.845295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.845244 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql"] Oct 30 17:11:36.846164 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846141 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a522dd46-312d-47ac-8a11-07a76546094a" containerName="maas-api" Oct 30 17:11:36.846164 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846165 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a522dd46-312d-47ac-8a11-07a76546094a" containerName="maas-api" Oct 30 17:11:36.846352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846192 6400 cpu_manager.go:414] "containerMap: removing container" podUID="aa869356-749f-4c10-b350-f1aa6fc87baf" containerName="payload-pre-processing" Oct 30 17:11:36.846352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846200 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa869356-749f-4c10-b350-f1aa6fc87baf" containerName="payload-pre-processing" Oct 30 17:11:36.846352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846223 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f12e857f-ab74-4122-abe5-5c0297070fc0" containerName="payload-processing" Oct 30 17:11:36.846352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846230 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f12e857f-ab74-4122-abe5-5c0297070fc0" containerName="payload-processing" Oct 30 17:11:36.846352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846281 6400 cpu_manager.go:414] "containerMap: removing container" podUID="7a348964-7aa9-4ca7-98b2-62992f766d4b" containerName="maas-api" Oct 30 17:11:36.846352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846290 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7a348964-7aa9-4ca7-98b2-62992f766d4b" containerName="maas-api" Oct 30 17:11:36.846352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846317 6400 cpu_manager.go:414] "containerMap: removing container" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" Oct 30 17:11:36.846352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846324 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" Oct 30 17:11:36.846775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846499 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa869356-749f-4c10-b350-f1aa6fc87baf" containerName="payload-pre-processing" Oct 30 17:11:36.846775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846516 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f12e857f-ab74-4122-abe5-5c0297070fc0" containerName="payload-processing" Oct 30 17:11:36.846775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846531 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7a348964-7aa9-4ca7-98b2-62992f766d4b" containerName="maas-api" Oct 30 17:11:36.846775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846544 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fc3c304-52a0-4f96-8f6b-021c866e3f0a" containerName="istio-proxy" Oct 30 17:11:36.846775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.846557 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a522dd46-312d-47ac-8a11-07a76546094a" containerName="maas-api" Oct 30 17:11:36.852295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.852275 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:36.870093 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.870068 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-4def7e-serving-cert\"" Oct 30 17:11:36.871016 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.870994 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql"] Oct 30 17:11:36.967670 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.967617 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfftr\" (UniqueName: \"kubernetes.io/projected/5cceafdb-df25-4512-8592-3aa8d575467c-kube-api-access-mfftr\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:36.967851 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.967759 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:36.967910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.967866 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cceafdb-df25-4512-8592-3aa8d575467c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:36.967979 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:36.967964 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-metrics-certs\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:37.069666 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:37.069587 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:37.070037 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:37.069731 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cceafdb-df25-4512-8592-3aa8d575467c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:37.070037 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:37.069944 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-metrics-certs\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:37.070283 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:37.070252 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfftr\" (UniqueName: \"kubernetes.io/projected/5cceafdb-df25-4512-8592-3aa8d575467c-kube-api-access-mfftr\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:37.070860 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:37.070831 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cceafdb-df25-4512-8592-3aa8d575467c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:37.073197 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:37.073174 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-metrics-certs\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:37.073292 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:37.073260 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:37.095115 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:37.095090 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfftr\" (UniqueName: \"kubernetes.io/projected/5cceafdb-df25-4512-8592-3aa8d575467c-kube-api-access-mfftr\") pod \"maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:37.168169 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:37.168080 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:37.476202 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:11:37.476171 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cceafdb_df25_4512_8592_3aa8d575467c.slice/crio-0e39f9e4f655366b86f17d35c56cdb9390d934b1fe4d69e3b881cc62d3b27f27 WatchSource:0}: Error finding container 0e39f9e4f655366b86f17d35c56cdb9390d934b1fe4d69e3b881cc62d3b27f27: Status 404 returned error can't find the container with id 0e39f9e4f655366b86f17d35c56cdb9390d934b1fe4d69e3b881cc62d3b27f27 Oct 30 17:11:37.476464 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:37.476445 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql"] Oct 30 17:11:38.478296 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:38.478261 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" event={"ID":"5cceafdb-df25-4512-8592-3aa8d575467c","Type":"ContainerStarted","Data":"9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1"} Oct 30 17:11:38.478734 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:38.478321 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" event={"ID":"5cceafdb-df25-4512-8592-3aa8d575467c","Type":"ContainerStarted","Data":"0e39f9e4f655366b86f17d35c56cdb9390d934b1fe4d69e3b881cc62d3b27f27"} Oct 30 17:11:38.478734 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:38.478360 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:38.521959 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:38.521913 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" podStartSLOduration=2.108766888 podStartE2EDuration="2.521898668s" totalImagesPullingTime="413.13178ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:38.521681503 +0000 UTC m=+3007.671962491" watchObservedRunningTime="2026-10-30 17:11:38.521898668 +0000 UTC m=+3007.672179656" Oct 30 17:11:41.284347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.284314 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p"] Oct 30 17:11:41.288968 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.288945 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.310686 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.310655 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p"] Oct 30 17:11:41.318831 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.318804 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-55e83e\"/\"default-dockercfg-svp88\"" Oct 30 17:11:41.318831 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.318817 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-55e83e\"/\"ipp-route-55e83e-kserve-self-signed-certs\"" Oct 30 17:11:41.319047 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.318949 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-55e83e\"/\"openshift-service-ca.crt\"" Oct 30 17:11:41.319047 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.318997 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-55e83e\"/\"kube-root-ca.crt\"" Oct 30 17:11:41.433125 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.433080 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-home\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.433125 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.433117 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-dshm\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.433405 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.433155 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7c05644-8e94-4ff9-a25e-127e568c1291-tls-certs\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.433405 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.433277 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-model-cache\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.433405 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.433359 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwn67\" (UniqueName: \"kubernetes.io/projected/c7c05644-8e94-4ff9-a25e-127e568c1291-kube-api-access-pwn67\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.433574 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.433451 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-tmp-dir\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.538840 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.538743 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-home\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.539547 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.539519 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-dshm\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.539763 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.539571 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7c05644-8e94-4ff9-a25e-127e568c1291-tls-certs\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.539895 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.539875 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-model-cache\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.539984 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.539461 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-home\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.540165 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.540145 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-model-cache\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.540260 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.540219 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwn67\" (UniqueName: \"kubernetes.io/projected/c7c05644-8e94-4ff9-a25e-127e568c1291-kube-api-access-pwn67\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.540336 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.540319 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-tmp-dir\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.540795 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.540776 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-tmp-dir\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.559310 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.559279 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7c05644-8e94-4ff9-a25e-127e568c1291-tls-certs\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.560313 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.560290 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-dshm\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.577403 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.577369 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwn67\" (UniqueName: \"kubernetes.io/projected/c7c05644-8e94-4ff9-a25e-127e568c1291-kube-api-access-pwn67\") pod \"ipp-route-55e83e-kserve-6d6895dd69-hwl7p\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.605626 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.604159 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:41.933165 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:41.933120 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p"] Oct 30 17:11:41.933479 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:11:41.933455 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc7c05644_8e94_4ff9_a25e_127e568c1291.slice/crio-f6cde990737015fed668bb83752015e27fbe60cc08ad74c06a5482d11f03a54a WatchSource:0}: Error finding container f6cde990737015fed668bb83752015e27fbe60cc08ad74c06a5482d11f03a54a: Status 404 returned error can't find the container with id f6cde990737015fed668bb83752015e27fbe60cc08ad74c06a5482d11f03a54a Oct 30 17:11:42.512083 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:42.512040 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" event={"ID":"c7c05644-8e94-4ff9-a25e-127e568c1291","Type":"ContainerStarted","Data":"d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab"} Oct 30 17:11:42.512083 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:42.512088 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" event={"ID":"c7c05644-8e94-4ff9-a25e-127e568c1291","Type":"ContainerStarted","Data":"f6cde990737015fed668bb83752015e27fbe60cc08ad74c06a5482d11f03a54a"} Oct 30 17:11:42.556796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:42.556731 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" podStartSLOduration=1.5567142299999999 podStartE2EDuration="1.55671423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:11:42.55482133 +0000 UTC m=+3011.705102323" watchObservedRunningTime="2026-10-30 17:11:42.55671423 +0000 UTC m=+3011.706995223" Oct 30 17:11:42.606337 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:42.606304 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:44.490744 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:44.490712 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:11:51.605933 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:51.605903 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:51.619343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:51.619318 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:52.598012 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:52.597983 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:11:54.601516 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:11:54.601476 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:11:58.599763 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.599732 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4"] Oct 30 17:11:58.604574 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.604551 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.624203 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.624176 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4"] Oct 30 17:11:58.632906 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.632854 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-0ae3fe-serving-cert\"" Oct 30 17:11:58.701573 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.701545 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.701776 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.701752 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.701866 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.701799 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.701953 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.701934 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pp5s2\" (UniqueName: \"kubernetes.io/projected/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-kube-api-access-pp5s2\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.804159 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.804123 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.804377 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.804201 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.804377 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.804337 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pp5s2\" (UniqueName: \"kubernetes.io/projected/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-kube-api-access-pp5s2\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.804512 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:11:58.804372 6400 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-serving-cert: secret "maas-api-e2e-ipp-b-0ae3fe-serving-cert" not found Oct 30 17:11:58.804512 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:11:58.804448 6400 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-maas-api-tls podName:8978fa9f-b2c7-4132-a36f-69c9a4a9389c nodeName:}" failed. No retries permitted until 2026-10-30 17:11:59.304421659 +0000 UTC m=+3028.454702643 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-maas-api-tls") pod "maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" (UID: "8978fa9f-b2c7-4132-a36f-69c9a4a9389c") : secret "maas-api-e2e-ipp-b-0ae3fe-serving-cert" not found Oct 30 17:11:58.804652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.804520 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.805209 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.805183 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.807386 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.807364 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-metrics-certs\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:58.829584 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:58.829549 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pp5s2\" (UniqueName: \"kubernetes.io/projected/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-kube-api-access-pp5s2\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:59.314441 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:59.314411 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:59.317083 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:59.317054 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-maas-api-tls\") pod \"maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:59.520726 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:59.520688 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:11:59.828525 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:11:59.828491 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8978fa9f_b2c7_4132_a36f_69c9a4a9389c.slice/crio-0f95dd0a32fee0132936c1873eacc3bd3e16192928dded1918070b02d28dae47 WatchSource:0}: Error finding container 0f95dd0a32fee0132936c1873eacc3bd3e16192928dded1918070b02d28dae47: Status 404 returned error can't find the container with id 0f95dd0a32fee0132936c1873eacc3bd3e16192928dded1918070b02d28dae47 Oct 30 17:11:59.829562 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:11:59.829542 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4"] Oct 30 17:12:00.465795 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.465712 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx"] Oct 30 17:12:00.465978 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.465909 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" podUID="74f6c435-d09d-4327-9068-3f4ab0cb41f2" containerName="maas-api" containerID="cri-o://afaec0b8b4e6cc5e66705cce7629355136525b79b5f1218b49787bafc119dc93" gracePeriod=30 Oct 30 17:12:00.654343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.654294 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" podUID="74f6c435-d09d-4327-9068-3f4ab0cb41f2" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.95:8443/health\": dial tcp 10.134.0.95:8443: connect: connection refused" Oct 30 17:12:00.842036 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.841767 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" event={"ID":"8978fa9f-b2c7-4132-a36f-69c9a4a9389c","Type":"ContainerStarted","Data":"10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652"} Oct 30 17:12:00.842036 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.841879 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" event={"ID":"8978fa9f-b2c7-4132-a36f-69c9a4a9389c","Type":"ContainerStarted","Data":"0f95dd0a32fee0132936c1873eacc3bd3e16192928dded1918070b02d28dae47"} Oct 30 17:12:00.843170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.842931 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:12:00.843746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.843724 6400 generic.go:356] "Generic (PLEG): container finished" podID="74f6c435-d09d-4327-9068-3f4ab0cb41f2" containerID="afaec0b8b4e6cc5e66705cce7629355136525b79b5f1218b49787bafc119dc93" exitCode=0 Oct 30 17:12:00.843997 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.843795 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" event={"ID":"74f6c435-d09d-4327-9068-3f4ab0cb41f2","Type":"ContainerDied","Data":"afaec0b8b4e6cc5e66705cce7629355136525b79b5f1218b49787bafc119dc93"} Oct 30 17:12:00.874086 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.874052 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 17:12:00.892878 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.892803 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" podStartSLOduration=2.521915913 podStartE2EDuration="2.892785647s" totalImagesPullingTime="370.869734ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:11:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:12:00.889646391 +0000 UTC m=+3030.039927379" watchObservedRunningTime="2026-10-30 17:12:00.892785647 +0000 UTC m=+3030.043066636" Oct 30 17:12:00.929695 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.929255 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-maas-api-tls\") pod \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " Oct 30 17:12:00.929695 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.929333 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74f6c435-d09d-4327-9068-3f4ab0cb41f2-kube-api-access-xf99h\" (UniqueName: \"kubernetes.io/projected/74f6c435-d09d-4327-9068-3f4ab0cb41f2-kube-api-access-xf99h\") pod \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " Oct 30 17:12:00.929695 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.929413 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74f6c435-d09d-4327-9068-3f4ab0cb41f2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f6c435-d09d-4327-9068-3f4ab0cb41f2-openshift-service-ca\") pod \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " Oct 30 17:12:00.930020 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.929742 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-metrics-certs\") pod \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\" (UID: \"74f6c435-d09d-4327-9068-3f4ab0cb41f2\") " Oct 30 17:12:00.953401 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.932271 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74f6c435-d09d-4327-9068-3f4ab0cb41f2-openshift-service-ca" pod "74f6c435-d09d-4327-9068-3f4ab0cb41f2" (UID: "74f6c435-d09d-4327-9068-3f4ab0cb41f2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:00.966770 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.954865 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-metrics-certs" pod "74f6c435-d09d-4327-9068-3f4ab0cb41f2" (UID: "74f6c435-d09d-4327-9068-3f4ab0cb41f2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:12:00.985475 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.985420 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f6c435-d09d-4327-9068-3f4ab0cb41f2-kube-api-access-xf99h" pod "74f6c435-d09d-4327-9068-3f4ab0cb41f2" (UID: "74f6c435-d09d-4327-9068-3f4ab0cb41f2"). InnerVolumeSpecName "kube-api-access-xf99h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:00.996975 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:00.993749 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-maas-api-tls" pod "74f6c435-d09d-4327-9068-3f4ab0cb41f2" (UID: "74f6c435-d09d-4327-9068-3f4ab0cb41f2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:12:01.032240 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:01.032192 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xf99h\" (UniqueName: \"kubernetes.io/projected/74f6c435-d09d-4327-9068-3f4ab0cb41f2-kube-api-access-xf99h\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:01.032240 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:01.032228 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/74f6c435-d09d-4327-9068-3f4ab0cb41f2-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:01.032240 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:01.032244 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:01.032553 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:01.032257 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/74f6c435-d09d-4327-9068-3f4ab0cb41f2-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:01.854663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:01.854588 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" Oct 30 17:12:01.854663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:01.854651 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx" event={"ID":"74f6c435-d09d-4327-9068-3f4ab0cb41f2","Type":"ContainerDied","Data":"5609248898f34c99c98e621b9564e8fb73c094374150491943dfeffb5419c955"} Oct 30 17:12:01.855179 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:01.854714 6400 scope.go:122] "RemoveContainer" containerID="afaec0b8b4e6cc5e66705cce7629355136525b79b5f1218b49787bafc119dc93" Oct 30 17:12:01.916066 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:01.916036 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx"] Oct 30 17:12:01.923323 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:01.923292 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-e43400-55f7667bbb-n4rmx"] Oct 30 17:12:02.969006 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:02.968969 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74f6c435-d09d-4327-9068-3f4ab0cb41f2" path="/var/lib/kubelet/pods/74f6c435-d09d-4327-9068-3f4ab0cb41f2/volumes" Oct 30 17:12:06.861673 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:06.861643 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:12:24.600588 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:12:24.600548 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:12:30.221456 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:30.221418 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr"] Oct 30 17:12:30.222050 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:30.221685 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" podUID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" containerName="istio-proxy" containerID="cri-o://6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347" gracePeriod=30 Oct 30 17:12:35.522006 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.521974 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:12:35.551383 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551350 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-podinfo\") pod \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " Oct 30 17:12:35.551577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551462 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istiod-ca-cert\") pod \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " Oct 30 17:12:35.551577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551491 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-data\") pod \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " Oct 30 17:12:35.551577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551569 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-socket\") pod \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " Oct 30 17:12:35.551810 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551634 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-certs\") pod \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " Oct 30 17:12:35.551810 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551674 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-kube-api-access-zsqsn\" (UniqueName: \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-kube-api-access-zsqsn\") pod \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " Oct 30 17:12:35.551810 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551710 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-envoy\") pod \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " Oct 30 17:12:35.551810 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551749 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-credential-socket\") pod \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " Oct 30 17:12:35.552035 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551827 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-token\" (UniqueName: \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-token\") pod \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\" (UID: \"3aae2a95-9ae0-456d-8115-8084d9dc7ab6\") " Oct 30 17:12:35.552035 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551854 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-data" pod "3aae2a95-9ae0-456d-8115-8084d9dc7ab6" (UID: "3aae2a95-9ae0-456d-8115-8084d9dc7ab6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:35.552035 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551958 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-certs" pod "3aae2a95-9ae0-456d-8115-8084d9dc7ab6" (UID: "3aae2a95-9ae0-456d-8115-8084d9dc7ab6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:35.552035 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.551996 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istiod-ca-cert" pod "3aae2a95-9ae0-456d-8115-8084d9dc7ab6" (UID: "3aae2a95-9ae0-456d-8115-8084d9dc7ab6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:12:35.552269 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.552164 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-socket" pod "3aae2a95-9ae0-456d-8115-8084d9dc7ab6" (UID: "3aae2a95-9ae0-456d-8115-8084d9dc7ab6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:35.552390 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.552368 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-credential-socket" pod "3aae2a95-9ae0-456d-8115-8084d9dc7ab6" (UID: "3aae2a95-9ae0-456d-8115-8084d9dc7ab6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:35.552581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.552538 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:35.552581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.552567 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:35.552581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.552580 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:35.552581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.552593 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:35.552581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.552649 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:35.554384 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.554337 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-envoy" pod "3aae2a95-9ae0-456d-8115-8084d9dc7ab6" (UID: "3aae2a95-9ae0-456d-8115-8084d9dc7ab6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:12:35.554384 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.554354 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-kube-api-access-zsqsn" pod "3aae2a95-9ae0-456d-8115-8084d9dc7ab6" (UID: "3aae2a95-9ae0-456d-8115-8084d9dc7ab6"). InnerVolumeSpecName "kube-api-access-zsqsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:35.554523 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.554413 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-podinfo" pod "3aae2a95-9ae0-456d-8115-8084d9dc7ab6" (UID: "3aae2a95-9ae0-456d-8115-8084d9dc7ab6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:12:35.554677 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.554660 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-token" pod "3aae2a95-9ae0-456d-8115-8084d9dc7ab6" (UID: "3aae2a95-9ae0-456d-8115-8084d9dc7ab6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:12:35.654202 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.654168 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:35.654202 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.654202 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:35.654416 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.654217 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:35.654416 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:35.654231 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsqsn\" (UniqueName: \"kubernetes.io/projected/3aae2a95-9ae0-456d-8115-8084d9dc7ab6-kube-api-access-zsqsn\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:12:36.104700 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:36.104661 6400 generic.go:356] "Generic (PLEG): container finished" podID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" containerID="6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347" exitCode=0 Oct 30 17:12:36.104912 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:36.104713 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" Oct 30 17:12:36.104912 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:36.104742 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" event={"ID":"3aae2a95-9ae0-456d-8115-8084d9dc7ab6","Type":"ContainerDied","Data":"6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347"} Oct 30 17:12:36.104912 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:36.104800 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr" event={"ID":"3aae2a95-9ae0-456d-8115-8084d9dc7ab6","Type":"ContainerDied","Data":"60b64b89555d3f119db9486068a71d830635b22c38335750717c407e46f75bdf"} Oct 30 17:12:36.104912 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:36.104826 6400 scope.go:122] "RemoveContainer" containerID="6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347" Oct 30 17:12:36.121492 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:36.121471 6400 scope.go:122] "RemoveContainer" containerID="6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347" Oct 30 17:12:36.121903 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:12:36.121881 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347\": container with ID starting with 6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347 not found: ID does not exist" containerID="6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347" Oct 30 17:12:36.121973 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:36.121918 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347"} err="failed to get container status \"6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347\": rpc error: code = NotFound desc = could not find container \"6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347\": container with ID starting with 6381fcffab76c4d2b785f0ebf8bdf62e6e4acb3a07d746f0e95850e67a410347 not found: ID does not exist" Oct 30 17:12:36.168748 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:36.168718 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr"] Oct 30 17:12:36.180176 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:36.180144 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-7c4b84-openshift-default-6774495b7c-b6vnr"] Oct 30 17:12:36.968857 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:12:36.968823 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" path="/var/lib/kubelet/pods/3aae2a95-9ae0-456d-8115-8084d9dc7ab6/volumes" Oct 30 17:12:54.602908 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:12:54.602871 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:13:20.644507 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:20.644476 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk_133c0bcd-04a0-410e-aafc-73c4279bd13f/payload-processing/0.log" Oct 30 17:13:23.978456 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:23.978416 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x"] Oct 30 17:13:23.988052 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:23.988011 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j"] Oct 30 17:13:24.602859 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:13:24.602820 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:13:27.544221 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:27.544184 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt"] Oct 30 17:13:27.544721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:27.544443 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" podUID="d4d0a60d-f6d5-4322-89cf-4c226de2b2e0" containerName="maas-api" containerID="cri-o://32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018" gracePeriod=30 Oct 30 17:13:27.913261 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:27.913230 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:13:28.034802 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.034766 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-maas-api-tls\") pod \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " Oct 30 17:13:28.035002 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.034908 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-kube-api-access-kkcs2\" (UniqueName: \"kubernetes.io/projected/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-kube-api-access-kkcs2\") pod \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " Oct 30 17:13:28.035002 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.034956 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-openshift-service-ca\") pod \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " Oct 30 17:13:28.035002 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.034998 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-metrics-certs\") pod \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\" (UID: \"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0\") " Oct 30 17:13:28.035383 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.035356 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-openshift-service-ca" pod "d4d0a60d-f6d5-4322-89cf-4c226de2b2e0" (UID: "d4d0a60d-f6d5-4322-89cf-4c226de2b2e0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:28.035589 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.035573 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:28.037273 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.037252 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-metrics-certs" pod "d4d0a60d-f6d5-4322-89cf-4c226de2b2e0" (UID: "d4d0a60d-f6d5-4322-89cf-4c226de2b2e0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:28.037716 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.037692 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-maas-api-tls" pod "d4d0a60d-f6d5-4322-89cf-4c226de2b2e0" (UID: "d4d0a60d-f6d5-4322-89cf-4c226de2b2e0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:28.037716 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.037699 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-kube-api-access-kkcs2" pod "d4d0a60d-f6d5-4322-89cf-4c226de2b2e0" (UID: "d4d0a60d-f6d5-4322-89cf-4c226de2b2e0"). InnerVolumeSpecName "kube-api-access-kkcs2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:28.160705 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.160619 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:28.160705 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.160646 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:28.160705 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.160659 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkcs2\" (UniqueName: \"kubernetes.io/projected/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0-kube-api-access-kkcs2\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:28.550314 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.550277 6400 generic.go:356] "Generic (PLEG): container finished" podID="d4d0a60d-f6d5-4322-89cf-4c226de2b2e0" containerID="32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018" exitCode=0 Oct 30 17:13:28.550792 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.550321 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" Oct 30 17:13:28.550792 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.550320 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" event={"ID":"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0","Type":"ContainerDied","Data":"32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018"} Oct 30 17:13:28.550792 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.550450 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt" event={"ID":"d4d0a60d-f6d5-4322-89cf-4c226de2b2e0","Type":"ContainerDied","Data":"e77a6daedf1c93f2bfa8639fa8634ae41d6acafc9d5b33db39849a8927e2045a"} Oct 30 17:13:28.550792 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.550478 6400 scope.go:122] "RemoveContainer" containerID="32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018" Oct 30 17:13:28.562647 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.562626 6400 scope.go:122] "RemoveContainer" containerID="32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018" Oct 30 17:13:28.562904 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:13:28.562886 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018\": container with ID starting with 32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018 not found: ID does not exist" containerID="32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018" Oct 30 17:13:28.562958 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.562912 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018"} err="failed to get container status \"32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018\": rpc error: code = NotFound desc = could not find container \"32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018\": container with ID starting with 32606bf1affa5df6a39c503e3fba63da89df19b79c000fa561809133483d5018 not found: ID does not exist" Oct 30 17:13:28.602591 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.602557 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt"] Oct 30 17:13:28.613484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.613456 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-330282-85c7b77ff5-rwxnt"] Oct 30 17:13:28.968784 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:28.968707 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4d0a60d-f6d5-4322-89cf-4c226de2b2e0" path="/var/lib/kubelet/pods/d4d0a60d-f6d5-4322-89cf-4c226de2b2e0/volumes" Oct 30 17:13:39.002645 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.002592 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" containerID="cri-o://d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c" gracePeriod=45 Oct 30 17:13:39.017062 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.017012 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" containerID="cri-o://584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250" gracePeriod=45 Oct 30 17:13:39.040674 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.040640 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" probeResult="failure" output="Get \"https://10.134.0.55:8000/ready\": dial tcp 10.134.0.55:8000: connect: connection refused" Oct 30 17:13:39.288035 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.288003 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 17:13:39.328505 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.328474 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421066 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-home\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-home\") pod \"3db653e1-2db6-485f-a401-df168bd9c785\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421132 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3db653e1-2db6-485f-a401-df168bd9c785-tls-certs\" (UniqueName: \"kubernetes.io/secret/3db653e1-2db6-485f-a401-df168bd9c785-tls-certs\") pod \"3db653e1-2db6-485f-a401-df168bd9c785\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421178 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-kube-api-access-hsthq\" (UniqueName: \"kubernetes.io/projected/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-kube-api-access-hsthq\") pod \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421224 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-dshm\") pod \"3db653e1-2db6-485f-a401-df168bd9c785\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421390 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-home\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-home\") pod \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421484 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-dshm\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-dshm\") pod \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421515 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3db653e1-2db6-485f-a401-df168bd9c785-kube-api-access-24x4q\" (UniqueName: \"kubernetes.io/projected/3db653e1-2db6-485f-a401-df168bd9c785-kube-api-access-24x4q\") pod \"3db653e1-2db6-485f-a401-df168bd9c785\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421547 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tmp-dir\") pod \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421588 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tls-certs\" (UniqueName: \"kubernetes.io/secret/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tls-certs\") pod \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421659 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-model-cache\") pod \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\" (UID: \"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421699 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-model-cache\") pod \"3db653e1-2db6-485f-a401-df168bd9c785\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " Oct 30 17:13:39.423177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.421816 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-tmp-dir\") pod \"3db653e1-2db6-485f-a401-df168bd9c785\" (UID: \"3db653e1-2db6-485f-a401-df168bd9c785\") " Oct 30 17:13:39.424087 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.424055 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-tmp-dir" pod "3db653e1-2db6-485f-a401-df168bd9c785" (UID: "3db653e1-2db6-485f-a401-df168bd9c785"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.425065 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.424780 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tmp-dir" pod "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" (UID: "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.432059 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.431827 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-model-cache" pod "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" (UID: "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.436191 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.433046 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-model-cache" pod "3db653e1-2db6-485f-a401-df168bd9c785" (UID: "3db653e1-2db6-485f-a401-df168bd9c785"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.436191 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.433153 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-home" pod "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" (UID: "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.436397 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.436243 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-home" pod "3db653e1-2db6-485f-a401-df168bd9c785" (UID: "3db653e1-2db6-485f-a401-df168bd9c785"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.437673 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.437646 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-dshm" pod "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" (UID: "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.437673 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.437660 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3db653e1-2db6-485f-a401-df168bd9c785-kube-api-access-24x4q" pod "3db653e1-2db6-485f-a401-df168bd9c785" (UID: "3db653e1-2db6-485f-a401-df168bd9c785"). InnerVolumeSpecName "kube-api-access-24x4q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:39.438752 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.438727 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-dshm" pod "3db653e1-2db6-485f-a401-df168bd9c785" (UID: "3db653e1-2db6-485f-a401-df168bd9c785"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:13:39.438752 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.438743 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-kube-api-access-hsthq" pod "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" (UID: "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03"). InnerVolumeSpecName "kube-api-access-hsthq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:39.439057 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.439037 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tls-certs" pod "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" (UID: "93bb0a4a-4dcb-49d6-98a5-eb537b02bc03"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:39.439123 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.439104 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3db653e1-2db6-485f-a401-df168bd9c785-tls-certs" pod "3db653e1-2db6-485f-a401-df168bd9c785" (UID: "3db653e1-2db6-485f-a401-df168bd9c785"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:39.524899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524818 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.524899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524843 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3db653e1-2db6-485f-a401-df168bd9c785-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.524899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524856 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsthq\" (UniqueName: \"kubernetes.io/projected/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-kube-api-access-hsthq\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.524899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524873 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.524899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524881 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.524899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524888 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.524899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524895 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24x4q\" (UniqueName: \"kubernetes.io/projected/3db653e1-2db6-485f-a401-df168bd9c785-kube-api-access-24x4q\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.524899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524903 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.525322 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524911 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.525322 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524918 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.525322 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524925 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.525322 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.524933 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3db653e1-2db6-485f-a401-df168bd9c785-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:39.645403 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.645370 6400 generic.go:356] "Generic (PLEG): container finished" podID="3db653e1-2db6-485f-a401-df168bd9c785" containerID="584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250" exitCode=0 Oct 30 17:13:39.645580 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.645432 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" Oct 30 17:13:39.645580 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.645454 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" event={"ID":"3db653e1-2db6-485f-a401-df168bd9c785","Type":"ContainerDied","Data":"584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250"} Oct 30 17:13:39.645580 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.645494 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j" event={"ID":"3db653e1-2db6-485f-a401-df168bd9c785","Type":"ContainerDied","Data":"4a79ce0c13d4cf13736ef2c0dc2d09d985cbae3ff0f66c26576c51a9cb6dcab9"} Oct 30 17:13:39.645580 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.645517 6400 scope.go:122] "RemoveContainer" containerID="584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250" Oct 30 17:13:39.647314 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.647294 6400 generic.go:356] "Generic (PLEG): container finished" podID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerID="d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c" exitCode=0 Oct 30 17:13:39.647438 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.647356 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" Oct 30 17:13:39.647438 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.647385 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" event={"ID":"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03","Type":"ContainerDied","Data":"d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c"} Oct 30 17:13:39.647562 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.647426 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x" event={"ID":"93bb0a4a-4dcb-49d6-98a5-eb537b02bc03","Type":"ContainerDied","Data":"4757cf9f2a5349d2833842bc575b4e91651e76cdf333560d2dda42c01f2fec0f"} Oct 30 17:13:39.661370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.661351 6400 scope.go:122] "RemoveContainer" containerID="584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250" Oct 30 17:13:39.661816 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:13:39.661778 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250\": container with ID starting with 584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250 not found: ID does not exist" containerID="584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250" Oct 30 17:13:39.661917 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.661820 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250"} err="failed to get container status \"584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250\": rpc error: code = NotFound desc = could not find container \"584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250\": container with ID starting with 584312d768d5233a5d15db66c15c415a2a0c450f9dfcaa128b765455eb089250 not found: ID does not exist" Oct 30 17:13:39.661917 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.661843 6400 scope.go:122] "RemoveContainer" containerID="d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c" Oct 30 17:13:39.674663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.674640 6400 scope.go:122] "RemoveContainer" containerID="d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c" Oct 30 17:13:39.674995 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:13:39.674962 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c\": container with ID starting with d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c not found: ID does not exist" containerID="d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c" Oct 30 17:13:39.675096 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.675003 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c"} err="failed to get container status \"d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c\": rpc error: code = NotFound desc = could not find container \"d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c\": container with ID starting with d54a03f090b3c8d1e8a0c459e4593182d22a8017ad3247e3a11d8350c442ba3c not found: ID does not exist" Oct 30 17:13:39.710467 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.710432 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j"] Oct 30 17:13:39.721674 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.721651 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/simulator-w5-967fac-kserve-6ccc68c76b-wqg5j"] Oct 30 17:13:39.754554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.754527 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x"] Oct 30 17:13:39.757738 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:39.757714 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-967fac/premium-w5-967fac-kserve-5d649b7b9f-xtm4x"] Oct 30 17:13:40.968962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:40.968927 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3db653e1-2db6-485f-a401-df168bd9c785" path="/var/lib/kubelet/pods/3db653e1-2db6-485f-a401-df168bd9c785/volumes" Oct 30 17:13:40.969347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:40.969336 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" path="/var/lib/kubelet/pods/93bb0a4a-4dcb-49d6-98a5-eb537b02bc03/volumes" Oct 30 17:13:46.447968 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.447935 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd"] Oct 30 17:13:46.448588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448569 6400 cpu_manager.go:414] "containerMap: removing container" podUID="74f6c435-d09d-4327-9068-3f4ab0cb41f2" containerName="maas-api" Oct 30 17:13:46.448588 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448585 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74f6c435-d09d-4327-9068-3f4ab0cb41f2" containerName="maas-api" Oct 30 17:13:46.448780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448623 6400 cpu_manager.go:414] "containerMap: removing container" podUID="d4d0a60d-f6d5-4322-89cf-4c226de2b2e0" containerName="maas-api" Oct 30 17:13:46.448780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448629 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4d0a60d-f6d5-4322-89cf-4c226de2b2e0" containerName="maas-api" Oct 30 17:13:46.448780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448645 6400 cpu_manager.go:414] "containerMap: removing container" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" Oct 30 17:13:46.448780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448650 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" Oct 30 17:13:46.448780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448675 6400 cpu_manager.go:414] "containerMap: removing container" podUID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" containerName="istio-proxy" Oct 30 17:13:46.448780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448680 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" containerName="istio-proxy" Oct 30 17:13:46.448780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448688 6400 cpu_manager.go:414] "containerMap: removing container" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" Oct 30 17:13:46.448780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448692 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" Oct 30 17:13:46.448780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448780 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4d0a60d-f6d5-4322-89cf-4c226de2b2e0" containerName="maas-api" Oct 30 17:13:46.449092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448790 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="93bb0a4a-4dcb-49d6-98a5-eb537b02bc03" containerName="main" Oct 30 17:13:46.449092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448801 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="3aae2a95-9ae0-456d-8115-8084d9dc7ab6" containerName="istio-proxy" Oct 30 17:13:46.449092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448810 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="3db653e1-2db6-485f-a401-df168bd9c785" containerName="main" Oct 30 17:13:46.449092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.448817 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="74f6c435-d09d-4327-9068-3f4ab0cb41f2" containerName="maas-api" Oct 30 17:13:46.453816 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.453796 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.474704 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.474675 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd"] Oct 30 17:13:46.481949 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.481921 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-ddc10d-serving-cert\"" Oct 30 17:13:46.532519 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.532488 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-metrics-certs\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.532717 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.532693 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffxcr\" (UniqueName: \"kubernetes.io/projected/641a877c-4c86-4e4c-b331-c8f71e8760a1-kube-api-access-ffxcr\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.532768 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.532738 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.532810 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.532766 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/641a877c-4c86-4e4c-b331-c8f71e8760a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.633926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.633884 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-metrics-certs\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.634139 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.634028 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffxcr\" (UniqueName: \"kubernetes.io/projected/641a877c-4c86-4e4c-b331-c8f71e8760a1-kube-api-access-ffxcr\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.634139 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.634073 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.634139 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.634100 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/641a877c-4c86-4e4c-b331-c8f71e8760a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.634781 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.634757 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/641a877c-4c86-4e4c-b331-c8f71e8760a1-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.636573 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.636553 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-maas-api-tls\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.636691 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.636653 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-metrics-certs\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.659516 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.659492 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffxcr\" (UniqueName: \"kubernetes.io/projected/641a877c-4c86-4e4c-b331-c8f71e8760a1-kube-api-access-ffxcr\") pod \"maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:46.769135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:46.769105 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:47.155650 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:13:47.151135 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod641a877c_4c86_4e4c_b331_c8f71e8760a1.slice/crio-d0e34a2f9dd31514076eb8edf56eba57e62cffd10902c432f4e8b925c9699ce7 WatchSource:0}: Error finding container d0e34a2f9dd31514076eb8edf56eba57e62cffd10902c432f4e8b925c9699ce7: Status 404 returned error can't find the container with id d0e34a2f9dd31514076eb8edf56eba57e62cffd10902c432f4e8b925c9699ce7 Oct 30 17:13:47.155650 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:47.151712 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd"] Oct 30 17:13:47.155650 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:47.153505 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:13:47.709892 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:47.709850 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" event={"ID":"641a877c-4c86-4e4c-b331-c8f71e8760a1","Type":"ContainerStarted","Data":"ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466"} Oct 30 17:13:47.710372 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:47.709911 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" event={"ID":"641a877c-4c86-4e4c-b331-c8f71e8760a1","Type":"ContainerStarted","Data":"d0e34a2f9dd31514076eb8edf56eba57e62cffd10902c432f4e8b925c9699ce7"} Oct 30 17:13:47.710372 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:47.710099 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:47.756737 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:47.756686 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" podStartSLOduration=1.342653429 podStartE2EDuration="1.756670381s" totalImagesPullingTime="414.016952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:13:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:13:47.753468497 +0000 UTC m=+3136.903749484" watchObservedRunningTime="2026-10-30 17:13:47.756670381 +0000 UTC m=+3136.906951370" Oct 30 17:13:53.721893 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:53.721859 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:13:54.601923 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:13:54.601887 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:13:55.680261 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:55.680224 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62"] Oct 30 17:13:55.680821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:55.680506 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" podUID="a8566fdd-4890-45e6-8654-d4ba5a0a9017" containerName="maas-api" containerID="cri-o://cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f" gracePeriod=30 Oct 30 17:13:55.715115 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:55.715067 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" podUID="a8566fdd-4890-45e6-8654-d4ba5a0a9017" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.169:8443/health\": dial tcp 10.134.0.169:8443: connect: connection refused" Oct 30 17:13:55.859905 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:55.858049 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4"] Oct 30 17:13:55.859905 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:55.858278 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" podUID="756535eb-8aac-4808-8a61-b2ac1b8813e2" containerName="payload-processing" containerID="cri-o://dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be" gracePeriod=30 Oct 30 17:13:55.885457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:55.885298 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66"] Oct 30 17:13:55.885651 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:55.885537 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" podUID="1f30a3fc-3120-441f-bb7b-717c3294e56c" containerName="payload-pre-processing" containerID="cri-o://e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4" gracePeriod=30 Oct 30 17:13:56.069597 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.069253 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:13:56.101623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.090132 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8566fdd-4890-45e6-8654-d4ba5a0a9017-kube-api-access-2spws\" (UniqueName: \"kubernetes.io/projected/a8566fdd-4890-45e6-8654-d4ba5a0a9017-kube-api-access-2spws\") pod \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " Oct 30 17:13:56.101623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.090226 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-metrics-certs\") pod \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " Oct 30 17:13:56.101623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.090440 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8566fdd-4890-45e6-8654-d4ba5a0a9017-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8566fdd-4890-45e6-8654-d4ba5a0a9017-openshift-service-ca\") pod \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " Oct 30 17:13:56.101623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.090535 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-maas-api-tls\") pod \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\" (UID: \"a8566fdd-4890-45e6-8654-d4ba5a0a9017\") " Oct 30 17:13:56.101623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.093236 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8566fdd-4890-45e6-8654-d4ba5a0a9017-openshift-service-ca" pod "a8566fdd-4890-45e6-8654-d4ba5a0a9017" (UID: "a8566fdd-4890-45e6-8654-d4ba5a0a9017"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:56.125066 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.125023 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8566fdd-4890-45e6-8654-d4ba5a0a9017-kube-api-access-2spws" pod "a8566fdd-4890-45e6-8654-d4ba5a0a9017" (UID: "a8566fdd-4890-45e6-8654-d4ba5a0a9017"). InnerVolumeSpecName "kube-api-access-2spws". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:56.128026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.127978 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-metrics-certs" pod "a8566fdd-4890-45e6-8654-d4ba5a0a9017" (UID: "a8566fdd-4890-45e6-8654-d4ba5a0a9017"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:56.131244 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.131210 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-maas-api-tls" pod "a8566fdd-4890-45e6-8654-d4ba5a0a9017" (UID: "a8566fdd-4890-45e6-8654-d4ba5a0a9017"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:13:56.193350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.193318 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8566fdd-4890-45e6-8654-d4ba5a0a9017-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:56.193350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.193349 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:56.193563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.193364 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2spws\" (UniqueName: \"kubernetes.io/projected/a8566fdd-4890-45e6-8654-d4ba5a0a9017-kube-api-access-2spws\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:56.193563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.193376 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8566fdd-4890-45e6-8654-d4ba5a0a9017-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:56.298819 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.298791 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:13:56.320192 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.320169 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:13:56.396703 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.396296 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f30a3fc-3120-441f-bb7b-717c3294e56c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f30a3fc-3120-441f-bb7b-717c3294e56c-plugins-config-volume\") pod \"1f30a3fc-3120-441f-bb7b-717c3294e56c\" (UID: \"1f30a3fc-3120-441f-bb7b-717c3294e56c\") " Oct 30 17:13:56.396703 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.396387 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f30a3fc-3120-441f-bb7b-717c3294e56c-kube-api-access-747tb\" (UniqueName: \"kubernetes.io/projected/1f30a3fc-3120-441f-bb7b-717c3294e56c-kube-api-access-747tb\") pod \"1f30a3fc-3120-441f-bb7b-717c3294e56c\" (UID: \"1f30a3fc-3120-441f-bb7b-717c3294e56c\") " Oct 30 17:13:56.396703 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.396526 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/756535eb-8aac-4808-8a61-b2ac1b8813e2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756535eb-8aac-4808-8a61-b2ac1b8813e2-plugins-config-volume\") pod \"756535eb-8aac-4808-8a61-b2ac1b8813e2\" (UID: \"756535eb-8aac-4808-8a61-b2ac1b8813e2\") " Oct 30 17:13:56.396703 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.396593 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/756535eb-8aac-4808-8a61-b2ac1b8813e2-kube-api-access-nlxb5\" (UniqueName: \"kubernetes.io/projected/756535eb-8aac-4808-8a61-b2ac1b8813e2-kube-api-access-nlxb5\") pod \"756535eb-8aac-4808-8a61-b2ac1b8813e2\" (UID: \"756535eb-8aac-4808-8a61-b2ac1b8813e2\") " Oct 30 17:13:56.418846 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.399490 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/756535eb-8aac-4808-8a61-b2ac1b8813e2-plugins-config-volume" pod "756535eb-8aac-4808-8a61-b2ac1b8813e2" (UID: "756535eb-8aac-4808-8a61-b2ac1b8813e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:56.418846 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.415393 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/756535eb-8aac-4808-8a61-b2ac1b8813e2-plugins-config-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:56.446455 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.419231 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f30a3fc-3120-441f-bb7b-717c3294e56c-plugins-config-volume" pod "1f30a3fc-3120-441f-bb7b-717c3294e56c" (UID: "1f30a3fc-3120-441f-bb7b-717c3294e56c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:13:56.446455 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.422050 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/756535eb-8aac-4808-8a61-b2ac1b8813e2-kube-api-access-nlxb5" pod "756535eb-8aac-4808-8a61-b2ac1b8813e2" (UID: "756535eb-8aac-4808-8a61-b2ac1b8813e2"). InnerVolumeSpecName "kube-api-access-nlxb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:56.473630 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.465947 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f30a3fc-3120-441f-bb7b-717c3294e56c-kube-api-access-747tb" pod "1f30a3fc-3120-441f-bb7b-717c3294e56c" (UID: "1f30a3fc-3120-441f-bb7b-717c3294e56c"). InnerVolumeSpecName "kube-api-access-747tb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:13:56.520434 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.520387 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f30a3fc-3120-441f-bb7b-717c3294e56c-plugins-config-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:56.520791 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.520766 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-747tb\" (UniqueName: \"kubernetes.io/projected/1f30a3fc-3120-441f-bb7b-717c3294e56c-kube-api-access-747tb\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:56.520972 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.520954 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlxb5\" (UniqueName: \"kubernetes.io/projected/756535eb-8aac-4808-8a61-b2ac1b8813e2-kube-api-access-nlxb5\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:13:56.783885 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.783799 6400 generic.go:356] "Generic (PLEG): container finished" podID="756535eb-8aac-4808-8a61-b2ac1b8813e2" containerID="dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be" exitCode=0 Oct 30 17:13:56.783885 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.783860 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" Oct 30 17:13:56.784420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.783880 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" event={"ID":"756535eb-8aac-4808-8a61-b2ac1b8813e2","Type":"ContainerDied","Data":"dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be"} Oct 30 17:13:56.784420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.783954 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4" event={"ID":"756535eb-8aac-4808-8a61-b2ac1b8813e2","Type":"ContainerDied","Data":"9850da1bfbd2683d18a25e78f81d4ff822c405b849df1c423d1aacfa838b982e"} Oct 30 17:13:56.784420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.783984 6400 scope.go:122] "RemoveContainer" containerID="dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be" Oct 30 17:13:56.785884 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.785846 6400 generic.go:356] "Generic (PLEG): container finished" podID="1f30a3fc-3120-441f-bb7b-717c3294e56c" containerID="e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4" exitCode=0 Oct 30 17:13:56.786119 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.786105 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" Oct 30 17:13:56.786397 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.786370 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" event={"ID":"1f30a3fc-3120-441f-bb7b-717c3294e56c","Type":"ContainerDied","Data":"e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4"} Oct 30 17:13:56.786454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.786433 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66" event={"ID":"1f30a3fc-3120-441f-bb7b-717c3294e56c","Type":"ContainerDied","Data":"254ac45fa3aa6e49dd800c00b50c1f11f3beee9dc963463045ad8eefa4171233"} Oct 30 17:13:56.792697 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.792669 6400 generic.go:356] "Generic (PLEG): container finished" podID="a8566fdd-4890-45e6-8654-d4ba5a0a9017" containerID="cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f" exitCode=0 Oct 30 17:13:56.792801 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.792734 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" Oct 30 17:13:56.792801 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.792753 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" event={"ID":"a8566fdd-4890-45e6-8654-d4ba5a0a9017","Type":"ContainerDied","Data":"cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f"} Oct 30 17:13:56.792801 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.792793 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62" event={"ID":"a8566fdd-4890-45e6-8654-d4ba5a0a9017","Type":"ContainerDied","Data":"c01e183794a45dda0bbbe08883f846ab7f9f51992cfb3f1922dc85bdb84dfab4"} Oct 30 17:13:56.799108 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.799088 6400 scope.go:122] "RemoveContainer" containerID="dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be" Oct 30 17:13:56.799584 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:13:56.799554 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be\": container with ID starting with dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be not found: ID does not exist" containerID="dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be" Oct 30 17:13:56.799713 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.799593 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be"} err="failed to get container status \"dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be\": rpc error: code = NotFound desc = could not find container \"dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be\": container with ID starting with dca2b0e6a0f32790397b0d5990b3ad32d5f107a853e9382b231f90985ee376be not found: ID does not exist" Oct 30 17:13:56.799713 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.799646 6400 scope.go:122] "RemoveContainer" containerID="e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4" Oct 30 17:13:56.817482 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.817453 6400 scope.go:122] "RemoveContainer" containerID="e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4" Oct 30 17:13:56.817872 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:13:56.817851 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4\": container with ID starting with e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4 not found: ID does not exist" containerID="e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4" Oct 30 17:13:56.817952 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.817882 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4"} err="failed to get container status \"e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4\": rpc error: code = NotFound desc = could not find container \"e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4\": container with ID starting with e7d746c68bbb128ee5d1bc94386921395e752965f38e0061a1ab745589477ef4 not found: ID does not exist" Oct 30 17:13:56.817952 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.817902 6400 scope.go:122] "RemoveContainer" containerID="cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f" Oct 30 17:13:56.833310 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.833288 6400 scope.go:122] "RemoveContainer" containerID="cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f" Oct 30 17:13:56.833670 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:13:56.833642 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f\": container with ID starting with cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f not found: ID does not exist" containerID="cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f" Oct 30 17:13:56.833738 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.833683 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f"} err="failed to get container status \"cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f\": rpc error: code = NotFound desc = could not find container \"cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f\": container with ID starting with cf5875659938658d6ca412433c4df08fd6e765520f7f7def7a1a9d599036aa7f not found: ID does not exist" Oct 30 17:13:56.879889 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.879853 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66"] Oct 30 17:13:56.883665 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.883640 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-e4048c-5665975cf-8ht66"] Oct 30 17:13:56.921704 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.921673 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4"] Oct 30 17:13:56.927222 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.927194 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-e4048c-bb5d69986-pnhs4"] Oct 30 17:13:56.970767 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.970733 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f30a3fc-3120-441f-bb7b-717c3294e56c" path="/var/lib/kubelet/pods/1f30a3fc-3120-441f-bb7b-717c3294e56c/volumes" Oct 30 17:13:56.971188 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.971173 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="756535eb-8aac-4808-8a61-b2ac1b8813e2" path="/var/lib/kubelet/pods/756535eb-8aac-4808-8a61-b2ac1b8813e2/volumes" Oct 30 17:13:56.971485 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.971470 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62"] Oct 30 17:13:56.971550 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:56.971500 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-e4048c-78db9f885c-5ps62"] Oct 30 17:13:58.969738 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:13:58.969705 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8566fdd-4890-45e6-8654-d4ba5a0a9017" path="/var/lib/kubelet/pods/a8566fdd-4890-45e6-8654-d4ba5a0a9017/volumes" Oct 30 17:14:13.160844 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:13.160753 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx"] Oct 30 17:14:13.161347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:13.161000 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" podUID="c339b975-ee73-4069-898e-1589b161c6cc" containerName="istio-proxy" containerID="cri-o://a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20" gracePeriod=30 Oct 30 17:14:17.268201 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:17.268158 6400 patch_prober.go:29] interesting pod/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:14:17.268653 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:17.268218 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" podUID="c339b975-ee73-4069-898e-1589b161c6cc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:14:18.469986 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.469957 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:14:18.526202 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526172 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-certs\") pod \"c339b975-ee73-4069-898e-1589b161c6cc\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " Oct 30 17:14:18.526360 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526249 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-socket\") pod \"c339b975-ee73-4069-898e-1589b161c6cc\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " Oct 30 17:14:18.526360 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526274 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-data\") pod \"c339b975-ee73-4069-898e-1589b161c6cc\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " Oct 30 17:14:18.526360 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526309 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-credential-socket\") pod \"c339b975-ee73-4069-898e-1589b161c6cc\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " Oct 30 17:14:18.526360 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526329 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c339b975-ee73-4069-898e-1589b161c6cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c339b975-ee73-4069-898e-1589b161c6cc-istiod-ca-cert\") pod \"c339b975-ee73-4069-898e-1589b161c6cc\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " Oct 30 17:14:18.526563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526389 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-kube-api-access-c4gqb\" (UniqueName: \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-kube-api-access-c4gqb\") pod \"c339b975-ee73-4069-898e-1589b161c6cc\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " Oct 30 17:14:18.526563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526445 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-istio-token\" (UniqueName: \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-istio-token\") pod \"c339b975-ee73-4069-898e-1589b161c6cc\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " Oct 30 17:14:18.526563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526478 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c339b975-ee73-4069-898e-1589b161c6cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c339b975-ee73-4069-898e-1589b161c6cc-istio-podinfo\") pod \"c339b975-ee73-4069-898e-1589b161c6cc\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " Oct 30 17:14:18.526563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526481 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-certs" pod "c339b975-ee73-4069-898e-1589b161c6cc" (UID: "c339b975-ee73-4069-898e-1589b161c6cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.526563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526551 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-socket" pod "c339b975-ee73-4069-898e-1589b161c6cc" (UID: "c339b975-ee73-4069-898e-1589b161c6cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.526969 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526562 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-credential-socket" pod "c339b975-ee73-4069-898e-1589b161c6cc" (UID: "c339b975-ee73-4069-898e-1589b161c6cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.526969 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526575 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-data" pod "c339b975-ee73-4069-898e-1589b161c6cc" (UID: "c339b975-ee73-4069-898e-1589b161c6cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.526969 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526689 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-envoy\") pod \"c339b975-ee73-4069-898e-1589b161c6cc\" (UID: \"c339b975-ee73-4069-898e-1589b161c6cc\") " Oct 30 17:14:18.526969 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.526861 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c339b975-ee73-4069-898e-1589b161c6cc-istiod-ca-cert" pod "c339b975-ee73-4069-898e-1589b161c6cc" (UID: "c339b975-ee73-4069-898e-1589b161c6cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:18.527363 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.527304 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.527363 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.527324 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.527363 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.527336 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.527363 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.527351 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.527363 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.527360 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c339b975-ee73-4069-898e-1589b161c6cc-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.529013 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.528989 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c339b975-ee73-4069-898e-1589b161c6cc-istio-podinfo" pod "c339b975-ee73-4069-898e-1589b161c6cc" (UID: "c339b975-ee73-4069-898e-1589b161c6cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:18.529162 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.529021 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-kube-api-access-c4gqb" pod "c339b975-ee73-4069-898e-1589b161c6cc" (UID: "c339b975-ee73-4069-898e-1589b161c6cc"). InnerVolumeSpecName "kube-api-access-c4gqb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:18.529355 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.529339 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-envoy" pod "c339b975-ee73-4069-898e-1589b161c6cc" (UID: "c339b975-ee73-4069-898e-1589b161c6cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:18.529355 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.529341 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-istio-token" pod "c339b975-ee73-4069-898e-1589b161c6cc" (UID: "c339b975-ee73-4069-898e-1589b161c6cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:18.629365 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.629322 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4gqb\" (UniqueName: \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-kube-api-access-c4gqb\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.629365 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.629370 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c339b975-ee73-4069-898e-1589b161c6cc-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.629660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.629386 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c339b975-ee73-4069-898e-1589b161c6cc-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.629660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.629400 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c339b975-ee73-4069-898e-1589b161c6cc-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:18.965788 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.965754 6400 generic.go:356] "Generic (PLEG): container finished" podID="c339b975-ee73-4069-898e-1589b161c6cc" containerID="a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20" exitCode=0 Oct 30 17:14:18.965970 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.965879 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" Oct 30 17:14:18.967954 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.967916 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" event={"ID":"c339b975-ee73-4069-898e-1589b161c6cc","Type":"ContainerDied","Data":"a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20"} Oct 30 17:14:18.968107 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.967968 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx" event={"ID":"c339b975-ee73-4069-898e-1589b161c6cc","Type":"ContainerDied","Data":"15db4b557c021cd7033662ae859b7549f96afe0aead568321438da63dcb6e011"} Oct 30 17:14:18.968107 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.967999 6400 scope.go:122] "RemoveContainer" containerID="a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20" Oct 30 17:14:18.980177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.980158 6400 scope.go:122] "RemoveContainer" containerID="a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20" Oct 30 17:14:18.980456 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:14:18.980436 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20\": container with ID starting with a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20 not found: ID does not exist" containerID="a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20" Oct 30 17:14:18.980558 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:18.980463 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20"} err="failed to get container status \"a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20\": rpc error: code = NotFound desc = could not find container \"a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20\": container with ID starting with a633559b33d240e2e4e6690dbc3982bc4a7a73a3143d3dcab02ae80a7a46be20 not found: ID does not exist" Oct 30 17:14:19.038828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:19.038799 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx"] Oct 30 17:14:19.051408 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:19.051371 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e4048c-openshift-default-5c764747c9-88zpx"] Oct 30 17:14:20.969361 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:20.969330 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c339b975-ee73-4069-898e-1589b161c6cc" path="/var/lib/kubelet/pods/c339b975-ee73-4069-898e-1589b161c6cc/volumes" Oct 30 17:14:24.601729 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:14:24.601686 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:14:25.126120 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.126090 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb"] Oct 30 17:14:25.126332 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.126303 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" podUID="d700bf64-2ff5-4546-92e1-c433fc8d3a61" containerName="maas-api" containerID="cri-o://277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c" gracePeriod=30 Oct 30 17:14:25.296474 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.296437 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk"] Oct 30 17:14:25.297017 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.296969 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" podUID="133c0bcd-04a0-410e-aafc-73c4279bd13f" containerName="payload-processing" containerID="cri-o://f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e" gracePeriod=30 Oct 30 17:14:25.466735 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.466694 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:14:25.588036 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.588012 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:14:25.640983 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.640950 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-metrics-certs\") pod \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " Oct 30 17:14:25.641483 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.640992 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d700bf64-2ff5-4546-92e1-c433fc8d3a61-kube-api-access-9dgww\" (UniqueName: \"kubernetes.io/projected/d700bf64-2ff5-4546-92e1-c433fc8d3a61-kube-api-access-9dgww\") pod \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " Oct 30 17:14:25.641483 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.641036 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-maas-api-tls\") pod \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " Oct 30 17:14:25.641483 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.641070 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d700bf64-2ff5-4546-92e1-c433fc8d3a61-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d700bf64-2ff5-4546-92e1-c433fc8d3a61-openshift-service-ca\") pod \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\" (UID: \"d700bf64-2ff5-4546-92e1-c433fc8d3a61\") " Oct 30 17:14:25.644179 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.644147 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-metrics-certs" pod "d700bf64-2ff5-4546-92e1-c433fc8d3a61" (UID: "d700bf64-2ff5-4546-92e1-c433fc8d3a61"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:25.644360 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.644264 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d700bf64-2ff5-4546-92e1-c433fc8d3a61-kube-api-access-9dgww" pod "d700bf64-2ff5-4546-92e1-c433fc8d3a61" (UID: "d700bf64-2ff5-4546-92e1-c433fc8d3a61"). InnerVolumeSpecName "kube-api-access-9dgww". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:25.644360 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.644279 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d700bf64-2ff5-4546-92e1-c433fc8d3a61-openshift-service-ca" pod "d700bf64-2ff5-4546-92e1-c433fc8d3a61" (UID: "d700bf64-2ff5-4546-92e1-c433fc8d3a61"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:25.644895 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.644875 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:25.645002 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.644900 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dgww\" (UniqueName: \"kubernetes.io/projected/d700bf64-2ff5-4546-92e1-c433fc8d3a61-kube-api-access-9dgww\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:25.645002 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.644916 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d700bf64-2ff5-4546-92e1-c433fc8d3a61-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:25.647200 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.647172 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-maas-api-tls" pod "d700bf64-2ff5-4546-92e1-c433fc8d3a61" (UID: "d700bf64-2ff5-4546-92e1-c433fc8d3a61"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:25.747623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.745651 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/133c0bcd-04a0-410e-aafc-73c4279bd13f-kube-api-access-lt4r6\" (UniqueName: \"kubernetes.io/projected/133c0bcd-04a0-410e-aafc-73c4279bd13f-kube-api-access-lt4r6\") pod \"133c0bcd-04a0-410e-aafc-73c4279bd13f\" (UID: \"133c0bcd-04a0-410e-aafc-73c4279bd13f\") " Oct 30 17:14:25.747623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.745833 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/133c0bcd-04a0-410e-aafc-73c4279bd13f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133c0bcd-04a0-410e-aafc-73c4279bd13f-plugins-config-volume\") pod \"133c0bcd-04a0-410e-aafc-73c4279bd13f\" (UID: \"133c0bcd-04a0-410e-aafc-73c4279bd13f\") " Oct 30 17:14:25.747623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.746209 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d700bf64-2ff5-4546-92e1-c433fc8d3a61-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:25.747623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.746524 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/133c0bcd-04a0-410e-aafc-73c4279bd13f-plugins-config-volume" pod "133c0bcd-04a0-410e-aafc-73c4279bd13f" (UID: "133c0bcd-04a0-410e-aafc-73c4279bd13f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:25.752634 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.751225 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/133c0bcd-04a0-410e-aafc-73c4279bd13f-kube-api-access-lt4r6" pod "133c0bcd-04a0-410e-aafc-73c4279bd13f" (UID: "133c0bcd-04a0-410e-aafc-73c4279bd13f"). InnerVolumeSpecName "kube-api-access-lt4r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:25.848135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.848102 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/133c0bcd-04a0-410e-aafc-73c4279bd13f-plugins-config-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:25.848135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:25.848134 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt4r6\" (UniqueName: \"kubernetes.io/projected/133c0bcd-04a0-410e-aafc-73c4279bd13f-kube-api-access-lt4r6\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:26.019783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.019690 6400 generic.go:356] "Generic (PLEG): container finished" podID="133c0bcd-04a0-410e-aafc-73c4279bd13f" containerID="f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e" exitCode=0 Oct 30 17:14:26.019783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.019737 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" Oct 30 17:14:26.020019 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.019775 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" event={"ID":"133c0bcd-04a0-410e-aafc-73c4279bd13f","Type":"ContainerDied","Data":"f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e"} Oct 30 17:14:26.020019 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.019838 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk" event={"ID":"133c0bcd-04a0-410e-aafc-73c4279bd13f","Type":"ContainerDied","Data":"5984eca59aa4fd228804ca388f1ff537f3ee62d376aa15856c7283119e92ec48"} Oct 30 17:14:26.020019 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.019865 6400 scope.go:122] "RemoveContainer" containerID="f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e" Oct 30 17:14:26.025308 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.025281 6400 generic.go:356] "Generic (PLEG): container finished" podID="d700bf64-2ff5-4546-92e1-c433fc8d3a61" containerID="277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c" exitCode=0 Oct 30 17:14:26.025437 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.025405 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" event={"ID":"d700bf64-2ff5-4546-92e1-c433fc8d3a61","Type":"ContainerDied","Data":"277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c"} Oct 30 17:14:26.025508 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.025451 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" event={"ID":"d700bf64-2ff5-4546-92e1-c433fc8d3a61","Type":"ContainerDied","Data":"6e80434cb7ee0bc3a63938e3d5ad00120409e744ec8904336bea5238a2c4f3ea"} Oct 30 17:14:26.025569 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.025528 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb" Oct 30 17:14:26.038719 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.038695 6400 scope.go:122] "RemoveContainer" containerID="f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e" Oct 30 17:14:26.039046 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:14:26.039024 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e\": container with ID starting with f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e not found: ID does not exist" containerID="f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e" Oct 30 17:14:26.039142 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.039052 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e"} err="failed to get container status \"f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e\": rpc error: code = NotFound desc = could not find container \"f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e\": container with ID starting with f9093740f4528ae0215932e74f72400b964776a64b2b2b3289f58dfefdd2dc3e not found: ID does not exist" Oct 30 17:14:26.039142 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.039071 6400 scope.go:122] "RemoveContainer" containerID="277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c" Oct 30 17:14:26.050962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.050944 6400 scope.go:122] "RemoveContainer" containerID="277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c" Oct 30 17:14:26.051301 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:14:26.051280 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c\": container with ID starting with 277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c not found: ID does not exist" containerID="277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c" Oct 30 17:14:26.051397 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.051306 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c"} err="failed to get container status \"277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c\": rpc error: code = NotFound desc = could not find container \"277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c\": container with ID starting with 277ba745923a56b92201d6985e6006f7784c2d71baa00f4d6b087f98ae51194c not found: ID does not exist" Oct 30 17:14:26.075709 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.075685 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk"] Oct 30 17:14:26.083879 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.083856 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-55e83e-8c6845fd8-z7bhk"] Oct 30 17:14:26.135428 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.135397 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb"] Oct 30 17:14:26.141350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.141319 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-55e83e-9758d54f4-2wtcb"] Oct 30 17:14:26.968167 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.968130 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="133c0bcd-04a0-410e-aafc-73c4279bd13f" path="/var/lib/kubelet/pods/133c0bcd-04a0-410e-aafc-73c4279bd13f/volumes" Oct 30 17:14:26.968615 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:26.968587 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d700bf64-2ff5-4546-92e1-c433fc8d3a61" path="/var/lib/kubelet/pods/d700bf64-2ff5-4546-92e1-c433fc8d3a61/volumes" Oct 30 17:14:31.369914 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:31.369878 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p"] Oct 30 17:14:43.957522 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:43.957480 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9"] Oct 30 17:14:43.958073 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:43.957816 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" podUID="40f37558-408c-4fea-a976-a3b2cb8bbe4d" containerName="maas-api" containerID="cri-o://a217cdca64d704fc3a84cc8ded18fa971f043ebeaa72e5f52552266e3e480152" gracePeriod=30 Oct 30 17:14:44.200679 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.198017 6400 generic.go:356] "Generic (PLEG): container finished" podID="40f37558-408c-4fea-a976-a3b2cb8bbe4d" containerID="a217cdca64d704fc3a84cc8ded18fa971f043ebeaa72e5f52552266e3e480152" exitCode=0 Oct 30 17:14:44.200679 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.198091 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" event={"ID":"40f37558-408c-4fea-a976-a3b2cb8bbe4d","Type":"ContainerDied","Data":"a217cdca64d704fc3a84cc8ded18fa971f043ebeaa72e5f52552266e3e480152"} Oct 30 17:14:44.280311 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.280277 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:14:44.423977 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.423939 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-maas-api-tls\") pod \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " Oct 30 17:14:44.424170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.424026 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40f37558-408c-4fea-a976-a3b2cb8bbe4d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f37558-408c-4fea-a976-a3b2cb8bbe4d-openshift-service-ca\") pod \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " Oct 30 17:14:44.424170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.424072 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40f37558-408c-4fea-a976-a3b2cb8bbe4d-kube-api-access-f2vxw\" (UniqueName: \"kubernetes.io/projected/40f37558-408c-4fea-a976-a3b2cb8bbe4d-kube-api-access-f2vxw\") pod \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " Oct 30 17:14:44.424308 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.424197 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-metrics-certs\") pod \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\" (UID: \"40f37558-408c-4fea-a976-a3b2cb8bbe4d\") " Oct 30 17:14:44.424461 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.424439 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40f37558-408c-4fea-a976-a3b2cb8bbe4d-openshift-service-ca" pod "40f37558-408c-4fea-a976-a3b2cb8bbe4d" (UID: "40f37558-408c-4fea-a976-a3b2cb8bbe4d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:44.424921 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.424844 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/40f37558-408c-4fea-a976-a3b2cb8bbe4d-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:44.426352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.426323 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-maas-api-tls" pod "40f37558-408c-4fea-a976-a3b2cb8bbe4d" (UID: "40f37558-408c-4fea-a976-a3b2cb8bbe4d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:44.426445 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.426378 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-metrics-certs" pod "40f37558-408c-4fea-a976-a3b2cb8bbe4d" (UID: "40f37558-408c-4fea-a976-a3b2cb8bbe4d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:44.426445 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.426416 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40f37558-408c-4fea-a976-a3b2cb8bbe4d-kube-api-access-f2vxw" pod "40f37558-408c-4fea-a976-a3b2cb8bbe4d" (UID: "40f37558-408c-4fea-a976-a3b2cb8bbe4d"). InnerVolumeSpecName "kube-api-access-f2vxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:44.526133 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.525807 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f2vxw\" (UniqueName: \"kubernetes.io/projected/40f37558-408c-4fea-a976-a3b2cb8bbe4d-kube-api-access-f2vxw\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:44.526133 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.525839 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:44.526133 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:44.525849 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/40f37558-408c-4fea-a976-a3b2cb8bbe4d-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:45.206932 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:45.206900 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" Oct 30 17:14:45.206932 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:45.206921 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9" event={"ID":"40f37558-408c-4fea-a976-a3b2cb8bbe4d","Type":"ContainerDied","Data":"d2226f12061818164ca48841ae435b8ba5f7ab84a26317bd0fcea9ca42857e89"} Oct 30 17:14:45.207440 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:45.206978 6400 scope.go:122] "RemoveContainer" containerID="a217cdca64d704fc3a84cc8ded18fa971f043ebeaa72e5f52552266e3e480152" Oct 30 17:14:45.262531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:45.262495 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9"] Oct 30 17:14:45.276325 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:45.276242 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-698c71-58758b8d94-xtwp9"] Oct 30 17:14:45.754420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:45.754382 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb"] Oct 30 17:14:45.754663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:45.754616 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" containerID="cri-o://4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7" gracePeriod=30 Oct 30 17:14:46.393533 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.393492 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" podUID="c7c05644-8e94-4ff9-a25e-127e568c1291" containerName="main" containerID="cri-o://d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab" gracePeriod=45 Oct 30 17:14:46.665275 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.665204 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:14:46.754255 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.754215 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7c05644-8e94-4ff9-a25e-127e568c1291-kube-api-access-pwn67\" (UniqueName: \"kubernetes.io/projected/c7c05644-8e94-4ff9-a25e-127e568c1291-kube-api-access-pwn67\") pod \"c7c05644-8e94-4ff9-a25e-127e568c1291\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " Oct 30 17:14:46.754460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.754276 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-dshm\") pod \"c7c05644-8e94-4ff9-a25e-127e568c1291\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " Oct 30 17:14:46.754460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.754419 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-model-cache\") pod \"c7c05644-8e94-4ff9-a25e-127e568c1291\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " Oct 30 17:14:46.754632 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.754537 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-tmp-dir\") pod \"c7c05644-8e94-4ff9-a25e-127e568c1291\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " Oct 30 17:14:46.754632 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.754579 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c7c05644-8e94-4ff9-a25e-127e568c1291-tls-certs\" (UniqueName: \"kubernetes.io/secret/c7c05644-8e94-4ff9-a25e-127e568c1291-tls-certs\") pod \"c7c05644-8e94-4ff9-a25e-127e568c1291\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " Oct 30 17:14:46.754764 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.754662 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-home\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-home\") pod \"c7c05644-8e94-4ff9-a25e-127e568c1291\" (UID: \"c7c05644-8e94-4ff9-a25e-127e568c1291\") " Oct 30 17:14:46.754882 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.754852 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-model-cache" pod "c7c05644-8e94-4ff9-a25e-127e568c1291" (UID: "c7c05644-8e94-4ff9-a25e-127e568c1291"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:46.755144 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.755119 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-home" pod "c7c05644-8e94-4ff9-a25e-127e568c1291" (UID: "c7c05644-8e94-4ff9-a25e-127e568c1291"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:46.755407 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.755383 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:46.755506 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.755409 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:46.755506 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.755411 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-tmp-dir" pod "c7c05644-8e94-4ff9-a25e-127e568c1291" (UID: "c7c05644-8e94-4ff9-a25e-127e568c1291"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:46.757162 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.757142 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7c05644-8e94-4ff9-a25e-127e568c1291-kube-api-access-pwn67" pod "c7c05644-8e94-4ff9-a25e-127e568c1291" (UID: "c7c05644-8e94-4ff9-a25e-127e568c1291"). InnerVolumeSpecName "kube-api-access-pwn67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:46.757320 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.757299 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-dshm" pod "c7c05644-8e94-4ff9-a25e-127e568c1291" (UID: "c7c05644-8e94-4ff9-a25e-127e568c1291"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:46.757572 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.757548 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c7c05644-8e94-4ff9-a25e-127e568c1291-tls-certs" pod "c7c05644-8e94-4ff9-a25e-127e568c1291" (UID: "c7c05644-8e94-4ff9-a25e-127e568c1291"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:46.856746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.856677 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwn67\" (UniqueName: \"kubernetes.io/projected/c7c05644-8e94-4ff9-a25e-127e568c1291-kube-api-access-pwn67\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:46.856746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.856714 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:46.856746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.856729 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c7c05644-8e94-4ff9-a25e-127e568c1291-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:46.856746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.856742 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c7c05644-8e94-4ff9-a25e-127e568c1291-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:46.967326 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:46.967242 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40f37558-408c-4fea-a976-a3b2cb8bbe4d" path="/var/lib/kubelet/pods/40f37558-408c-4fea-a976-a3b2cb8bbe4d/volumes" Oct 30 17:14:47.223194 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:47.223108 6400 generic.go:356] "Generic (PLEG): container finished" podID="c7c05644-8e94-4ff9-a25e-127e568c1291" containerID="d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab" exitCode=0 Oct 30 17:14:47.223194 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:47.223155 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" Oct 30 17:14:47.223403 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:47.223193 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" event={"ID":"c7c05644-8e94-4ff9-a25e-127e568c1291","Type":"ContainerDied","Data":"d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab"} Oct 30 17:14:47.223403 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:47.223235 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" event={"ID":"c7c05644-8e94-4ff9-a25e-127e568c1291","Type":"ContainerDied","Data":"f6cde990737015fed668bb83752015e27fbe60cc08ad74c06a5482d11f03a54a"} Oct 30 17:14:47.223403 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:47.223254 6400 scope.go:122] "RemoveContainer" containerID="d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab" Oct 30 17:14:47.233161 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:47.233136 6400 scope.go:122] "RemoveContainer" containerID="d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab" Oct 30 17:14:47.233445 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:14:47.233418 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab\": container with ID starting with d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab not found: ID does not exist" containerID="d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab" Oct 30 17:14:47.233493 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:47.233455 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab"} err="failed to get container status \"d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab\": rpc error: code = NotFound desc = could not find container \"d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab\": container with ID starting with d0473e71cb19d6aa3d8621ccd542194d72d6b037dbb65a9b0ea11f488fce29ab not found: ID does not exist" Oct 30 17:14:47.281949 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:47.281912 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p"] Oct 30 17:14:47.299776 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:47.299508 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p"] Oct 30 17:14:47.587423 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:47.587382 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-ipp-a-55e83e/ipp-route-55e83e-kserve-6d6895dd69-hwl7p" podUID="c7c05644-8e94-4ff9-a25e-127e568c1291" containerName="main" probeResult="failure" output="Get \"https://10.134.0.179:8000/ready\": context deadline exceeded" Oct 30 17:14:48.967480 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:48.967437 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7c05644-8e94-4ff9-a25e-127e568c1291" path="/var/lib/kubelet/pods/c7c05644-8e94-4ff9-a25e-127e568c1291/volumes" Oct 30 17:14:51.033819 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.033753 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:14:51.111309 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111272 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-credential-socket\") pod \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " Oct 30 17:14:51.111509 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111345 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-token\" (UniqueName: \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-token\") pod \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " Oct 30 17:14:51.111509 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111476 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-podinfo\") pod \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " Oct 30 17:14:51.111670 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111543 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-data\") pod \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " Oct 30 17:14:51.111670 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111581 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istiod-ca-cert\") pod \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " Oct 30 17:14:51.111670 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111629 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-certs\") pod \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " Oct 30 17:14:51.111894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111713 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-kube-api-access-nvbpj\" (UniqueName: \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-kube-api-access-nvbpj\") pod \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " Oct 30 17:14:51.111894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111806 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-socket\") pod \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " Oct 30 17:14:51.111894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111624 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-credential-socket" pod "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" (UID: "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:51.111894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111874 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-envoy\") pod \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\" (UID: \"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e\") " Oct 30 17:14:51.111894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111877 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-data" pod "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" (UID: "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:51.112152 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111894 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-certs" pod "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" (UID: "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:51.112152 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.111962 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istiod-ca-cert" pod "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" (UID: "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:51.112152 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.112122 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-socket" pod "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" (UID: "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:51.112829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.112647 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.112829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.112677 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.112829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.112692 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.112829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.112705 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.112829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.112718 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.114585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.114554 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-token" pod "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" (UID: "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:51.114971 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.114947 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-kube-api-access-nvbpj" pod "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" (UID: "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e"). InnerVolumeSpecName "kube-api-access-nvbpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:51.115108 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.115089 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-envoy" pod "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" (UID: "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:14:51.115298 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.115269 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-podinfo" pod "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" (UID: "6efd7c15-73f9-449c-b4fe-ffbb9123fa4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:14:51.214437 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.214400 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.214437 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.214431 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.214437 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.214441 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvbpj\" (UniqueName: \"kubernetes.io/projected/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-kube-api-access-nvbpj\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.214737 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.214449 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:51.252287 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.252257 6400 generic.go:356] "Generic (PLEG): container finished" podID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerID="4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7" exitCode=0 Oct 30 17:14:51.252446 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.252315 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" Oct 30 17:14:51.252446 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.252343 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" event={"ID":"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e","Type":"ContainerDied","Data":"4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7"} Oct 30 17:14:51.252446 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.252397 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb" event={"ID":"6efd7c15-73f9-449c-b4fe-ffbb9123fa4e","Type":"ContainerDied","Data":"f87f1c099410e073080d0b83b2af4c530bc6b4bd505343496f6141e115702092"} Oct 30 17:14:51.252446 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.252423 6400 scope.go:122] "RemoveContainer" containerID="4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7" Oct 30 17:14:51.264078 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.264059 6400 scope.go:122] "RemoveContainer" containerID="4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7" Oct 30 17:14:51.264351 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:14:51.264333 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7\": container with ID starting with 4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7 not found: ID does not exist" containerID="4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7" Oct 30 17:14:51.264400 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.264359 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7"} err="failed to get container status \"4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7\": rpc error: code = NotFound desc = could not find container \"4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7\": container with ID starting with 4ab1c01678a8a02e5a80f217e45d2bc2687b459d9abc1dd756c265f88bac59f7 not found: ID does not exist" Oct 30 17:14:51.309429 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.309366 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb"] Oct 30 17:14:51.322050 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:51.322024 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-bfecdd-openshift-default-6c8788cddc-lqtvb"] Oct 30 17:14:52.967779 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:52.967741 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" path="/var/lib/kubelet/pods/6efd7c15-73f9-449c-b4fe-ffbb9123fa4e/volumes" Oct 30 17:14:54.599686 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:14:54.599646 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:14:54.988424 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:54.988395 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4"] Oct 30 17:14:54.988646 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:54.988619 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" podUID="90d76b4b-e64a-4e0d-bcac-30a8104e2c20" containerName="maas-api" containerID="cri-o://cc1928b53bd13b2bf45c485111bdece3dbfedfb6e86309d6ba0b55adaac4b46c" gracePeriod=30 Oct 30 17:14:55.154265 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.154223 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2"] Oct 30 17:14:55.154535 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.154509 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" podUID="ddc37d3c-fb6d-4b17-94c3-1619c26349c5" containerName="payload-processing" containerID="cri-o://e6e03e333c76c73ff16be90e57ebca7a740251831a178331f6abcedd6e4cc931" gracePeriod=30 Oct 30 17:14:55.284678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.284549 6400 generic.go:356] "Generic (PLEG): container finished" podID="90d76b4b-e64a-4e0d-bcac-30a8104e2c20" containerID="cc1928b53bd13b2bf45c485111bdece3dbfedfb6e86309d6ba0b55adaac4b46c" exitCode=0 Oct 30 17:14:55.284678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.284638 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" event={"ID":"90d76b4b-e64a-4e0d-bcac-30a8104e2c20","Type":"ContainerDied","Data":"cc1928b53bd13b2bf45c485111bdece3dbfedfb6e86309d6ba0b55adaac4b46c"} Oct 30 17:14:55.287496 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.287427 6400 generic.go:356] "Generic (PLEG): container finished" podID="ddc37d3c-fb6d-4b17-94c3-1619c26349c5" containerID="e6e03e333c76c73ff16be90e57ebca7a740251831a178331f6abcedd6e4cc931" exitCode=0 Oct 30 17:14:55.287769 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.287509 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" event={"ID":"ddc37d3c-fb6d-4b17-94c3-1619c26349c5","Type":"ContainerDied","Data":"e6e03e333c76c73ff16be90e57ebca7a740251831a178331f6abcedd6e4cc931"} Oct 30 17:14:55.342427 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.342398 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:14:55.433297 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.433260 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql"] Oct 30 17:14:55.433553 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.433525 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" podUID="5cceafdb-df25-4512-8592-3aa8d575467c" containerName="maas-api" containerID="cri-o://9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1" gracePeriod=30 Oct 30 17:14:55.469740 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.469704 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-openshift-service-ca\") pod \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " Oct 30 17:14:55.469915 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.469828 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-metrics-certs\") pod \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " Oct 30 17:14:55.469915 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.469890 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-maas-api-tls\") pod \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " Oct 30 17:14:55.470010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.469950 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-kube-api-access-65zwh\" (UniqueName: \"kubernetes.io/projected/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-kube-api-access-65zwh\") pod \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\" (UID: \"90d76b4b-e64a-4e0d-bcac-30a8104e2c20\") " Oct 30 17:14:55.471882 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.471276 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-openshift-service-ca" pod "90d76b4b-e64a-4e0d-bcac-30a8104e2c20" (UID: "90d76b4b-e64a-4e0d-bcac-30a8104e2c20"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:55.474562 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.474528 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-metrics-certs" pod "90d76b4b-e64a-4e0d-bcac-30a8104e2c20" (UID: "90d76b4b-e64a-4e0d-bcac-30a8104e2c20"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:55.476809 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.476785 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-kube-api-access-65zwh" pod "90d76b4b-e64a-4e0d-bcac-30a8104e2c20" (UID: "90d76b4b-e64a-4e0d-bcac-30a8104e2c20"). InnerVolumeSpecName "kube-api-access-65zwh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:55.478827 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.478798 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-maas-api-tls" pod "90d76b4b-e64a-4e0d-bcac-30a8104e2c20" (UID: "90d76b4b-e64a-4e0d-bcac-30a8104e2c20"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:55.551453 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.551417 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:14:55.573627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.571998 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:55.573627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.572030 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65zwh\" (UniqueName: \"kubernetes.io/projected/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-kube-api-access-65zwh\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:55.573627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.572046 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:55.573627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.572060 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90d76b4b-e64a-4e0d-bcac-30a8104e2c20-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:55.683034 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.682993 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-plugins-config-volume\") pod \"ddc37d3c-fb6d-4b17-94c3-1619c26349c5\" (UID: \"ddc37d3c-fb6d-4b17-94c3-1619c26349c5\") " Oct 30 17:14:55.684053 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.684030 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-kube-api-access-b495q\" (UniqueName: \"kubernetes.io/projected/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-kube-api-access-b495q\") pod \"ddc37d3c-fb6d-4b17-94c3-1619c26349c5\" (UID: \"ddc37d3c-fb6d-4b17-94c3-1619c26349c5\") " Oct 30 17:14:55.684841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.683904 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-plugins-config-volume" pod "ddc37d3c-fb6d-4b17-94c3-1619c26349c5" (UID: "ddc37d3c-fb6d-4b17-94c3-1619c26349c5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:55.685490 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.685471 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-plugins-config-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:55.687299 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.687270 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-kube-api-access-b495q" pod "ddc37d3c-fb6d-4b17-94c3-1619c26349c5" (UID: "ddc37d3c-fb6d-4b17-94c3-1619c26349c5"). InnerVolumeSpecName "kube-api-access-b495q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:55.779947 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.779921 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:14:55.787027 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.787002 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b495q\" (UniqueName: \"kubernetes.io/projected/ddc37d3c-fb6d-4b17-94c3-1619c26349c5-kube-api-access-b495q\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:55.888351 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.888267 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cceafdb-df25-4512-8592-3aa8d575467c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cceafdb-df25-4512-8592-3aa8d575467c-openshift-service-ca\") pod \"5cceafdb-df25-4512-8592-3aa8d575467c\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " Oct 30 17:14:55.888351 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.888318 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-maas-api-tls\") pod \"5cceafdb-df25-4512-8592-3aa8d575467c\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " Oct 30 17:14:55.888746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.888407 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-metrics-certs\") pod \"5cceafdb-df25-4512-8592-3aa8d575467c\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " Oct 30 17:14:55.888746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.888432 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cceafdb-df25-4512-8592-3aa8d575467c-kube-api-access-mfftr\" (UniqueName: \"kubernetes.io/projected/5cceafdb-df25-4512-8592-3aa8d575467c-kube-api-access-mfftr\") pod \"5cceafdb-df25-4512-8592-3aa8d575467c\" (UID: \"5cceafdb-df25-4512-8592-3aa8d575467c\") " Oct 30 17:14:55.888746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.888724 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cceafdb-df25-4512-8592-3aa8d575467c-openshift-service-ca" pod "5cceafdb-df25-4512-8592-3aa8d575467c" (UID: "5cceafdb-df25-4512-8592-3aa8d575467c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:14:55.889034 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.889011 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cceafdb-df25-4512-8592-3aa8d575467c-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:55.890804 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.890779 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-maas-api-tls" pod "5cceafdb-df25-4512-8592-3aa8d575467c" (UID: "5cceafdb-df25-4512-8592-3aa8d575467c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:55.890905 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.890816 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-metrics-certs" pod "5cceafdb-df25-4512-8592-3aa8d575467c" (UID: "5cceafdb-df25-4512-8592-3aa8d575467c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:14:55.890978 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:55.890963 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cceafdb-df25-4512-8592-3aa8d575467c-kube-api-access-mfftr" pod "5cceafdb-df25-4512-8592-3aa8d575467c" (UID: "5cceafdb-df25-4512-8592-3aa8d575467c"). InnerVolumeSpecName "kube-api-access-mfftr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:14:56.028824 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.028797 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:56.028824 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.028824 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5cceafdb-df25-4512-8592-3aa8d575467c-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:56.029057 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.028838 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfftr\" (UniqueName: \"kubernetes.io/projected/5cceafdb-df25-4512-8592-3aa8d575467c-kube-api-access-mfftr\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:14:56.297419 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.297382 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" Oct 30 17:14:56.297419 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.297390 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4" event={"ID":"90d76b4b-e64a-4e0d-bcac-30a8104e2c20","Type":"ContainerDied","Data":"d77cff7fc16c59309f4f258a1d0d38d5629a8e4e4796790a67867153a95f400e"} Oct 30 17:14:56.297739 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.297453 6400 scope.go:122] "RemoveContainer" containerID="cc1928b53bd13b2bf45c485111bdece3dbfedfb6e86309d6ba0b55adaac4b46c" Oct 30 17:14:56.299008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.298987 6400 generic.go:356] "Generic (PLEG): container finished" podID="5cceafdb-df25-4512-8592-3aa8d575467c" containerID="9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1" exitCode=0 Oct 30 17:14:56.299120 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.299046 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" Oct 30 17:14:56.299120 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.299052 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" event={"ID":"5cceafdb-df25-4512-8592-3aa8d575467c","Type":"ContainerDied","Data":"9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1"} Oct 30 17:14:56.299120 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.299097 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql" event={"ID":"5cceafdb-df25-4512-8592-3aa8d575467c","Type":"ContainerDied","Data":"0e39f9e4f655366b86f17d35c56cdb9390d934b1fe4d69e3b881cc62d3b27f27"} Oct 30 17:14:56.300744 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.300727 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" Oct 30 17:14:56.300902 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.300790 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2" event={"ID":"ddc37d3c-fb6d-4b17-94c3-1619c26349c5","Type":"ContainerDied","Data":"e78019e4d76436eeccabe326bc97794cfda7319ec21bc54974c9474d23207774"} Oct 30 17:14:56.310348 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.310259 6400 scope.go:122] "RemoveContainer" containerID="9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1" Oct 30 17:14:56.320498 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.320480 6400 scope.go:122] "RemoveContainer" containerID="9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1" Oct 30 17:14:56.320807 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:14:56.320788 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1\": container with ID starting with 9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1 not found: ID does not exist" containerID="9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1" Oct 30 17:14:56.320851 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.320817 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1"} err="failed to get container status \"9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1\": rpc error: code = NotFound desc = could not find container \"9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1\": container with ID starting with 9519129606d13e3ffbbbc811fbe138b429b50801b2dbda36c73e5c56e3c8ddd1 not found: ID does not exist" Oct 30 17:14:56.320851 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.320835 6400 scope.go:122] "RemoveContainer" containerID="e6e03e333c76c73ff16be90e57ebca7a740251831a178331f6abcedd6e4cc931" Oct 30 17:14:56.356963 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.356937 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql"] Oct 30 17:14:56.370097 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.370074 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-4def7e-677bc7ccd9-9x4ql"] Oct 30 17:14:56.398548 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.398502 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2"] Oct 30 17:14:56.403817 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.403785 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4ac125-7cf5b68cb6-z6hv2"] Oct 30 17:14:56.440496 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.440462 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4"] Oct 30 17:14:56.446869 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.446837 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-4ac125-64d976f6b7-4bzg4"] Oct 30 17:14:56.968868 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.968835 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cceafdb-df25-4512-8592-3aa8d575467c" path="/var/lib/kubelet/pods/5cceafdb-df25-4512-8592-3aa8d575467c/volumes" Oct 30 17:14:56.969304 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.969290 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90d76b4b-e64a-4e0d-bcac-30a8104e2c20" path="/var/lib/kubelet/pods/90d76b4b-e64a-4e0d-bcac-30a8104e2c20/volumes" Oct 30 17:14:56.969702 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:14:56.969688 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddc37d3c-fb6d-4b17-94c3-1619c26349c5" path="/var/lib/kubelet/pods/ddc37d3c-fb6d-4b17-94c3-1619c26349c5/volumes" Oct 30 17:15:00.183229 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.183192 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t"] Oct 30 17:15:00.184047 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184023 6400 cpu_manager.go:414] "containerMap: removing container" podUID="d700bf64-2ff5-4546-92e1-c433fc8d3a61" containerName="maas-api" Oct 30 17:15:00.184047 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184047 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d700bf64-2ff5-4546-92e1-c433fc8d3a61" containerName="maas-api" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184062 6400 cpu_manager.go:414] "containerMap: removing container" podUID="90d76b4b-e64a-4e0d-bcac-30a8104e2c20" containerName="maas-api" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184069 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90d76b4b-e64a-4e0d-bcac-30a8104e2c20" containerName="maas-api" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184096 6400 cpu_manager.go:414] "containerMap: removing container" podUID="5cceafdb-df25-4512-8592-3aa8d575467c" containerName="maas-api" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184103 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cceafdb-df25-4512-8592-3aa8d575467c" containerName="maas-api" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184134 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c7c05644-8e94-4ff9-a25e-127e568c1291" containerName="main" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184142 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7c05644-8e94-4ff9-a25e-127e568c1291" containerName="main" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184162 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184169 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184181 6400 cpu_manager.go:414] "containerMap: removing container" podUID="756535eb-8aac-4808-8a61-b2ac1b8813e2" containerName="payload-processing" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184189 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="756535eb-8aac-4808-8a61-b2ac1b8813e2" containerName="payload-processing" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184209 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1f30a3fc-3120-441f-bb7b-717c3294e56c" containerName="payload-pre-processing" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184216 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f30a3fc-3120-441f-bb7b-717c3294e56c" containerName="payload-pre-processing" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184226 6400 cpu_manager.go:414] "containerMap: removing container" podUID="40f37558-408c-4fea-a976-a3b2cb8bbe4d" containerName="maas-api" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184233 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40f37558-408c-4fea-a976-a3b2cb8bbe4d" containerName="maas-api" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184256 6400 cpu_manager.go:414] "containerMap: removing container" podUID="133c0bcd-04a0-410e-aafc-73c4279bd13f" containerName="payload-processing" Oct 30 17:15:00.184266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184264 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="133c0bcd-04a0-410e-aafc-73c4279bd13f" containerName="payload-processing" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184291 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a8566fdd-4890-45e6-8654-d4ba5a0a9017" containerName="maas-api" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184298 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8566fdd-4890-45e6-8654-d4ba5a0a9017" containerName="maas-api" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184312 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ddc37d3c-fb6d-4b17-94c3-1619c26349c5" containerName="payload-processing" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184320 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddc37d3c-fb6d-4b17-94c3-1619c26349c5" containerName="payload-processing" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184338 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c339b975-ee73-4069-898e-1589b161c6cc" containerName="istio-proxy" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184345 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c339b975-ee73-4069-898e-1589b161c6cc" containerName="istio-proxy" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184492 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d700bf64-2ff5-4546-92e1-c433fc8d3a61" containerName="maas-api" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184507 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="90d76b4b-e64a-4e0d-bcac-30a8104e2c20" containerName="maas-api" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184518 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cceafdb-df25-4512-8592-3aa8d575467c" containerName="maas-api" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184531 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7c05644-8e94-4ff9-a25e-127e568c1291" containerName="main" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184544 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6efd7c15-73f9-449c-b4fe-ffbb9123fa4e" containerName="istio-proxy" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184553 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="756535eb-8aac-4808-8a61-b2ac1b8813e2" containerName="payload-processing" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184564 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f30a3fc-3120-441f-bb7b-717c3294e56c" containerName="payload-pre-processing" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184573 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="40f37558-408c-4fea-a976-a3b2cb8bbe4d" containerName="maas-api" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184584 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="133c0bcd-04a0-410e-aafc-73c4279bd13f" containerName="payload-processing" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184593 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8566fdd-4890-45e6-8654-d4ba5a0a9017" containerName="maas-api" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184621 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddc37d3c-fb6d-4b17-94c3-1619c26349c5" containerName="payload-processing" Oct 30 17:15:00.185129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.184632 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c339b975-ee73-4069-898e-1589b161c6cc" containerName="istio-proxy" Oct 30 17:15:00.190991 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.190969 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" Oct 30 17:15:00.211628 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.209680 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92"] Oct 30 17:15:00.217446 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.217417 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t"] Oct 30 17:15:00.217625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.217537 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" Oct 30 17:15:00.234627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.231485 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd"] Oct 30 17:15:00.242321 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.240451 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57"] Oct 30 17:15:00.242321 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.240580 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" Oct 30 17:15:00.248275 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.248242 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92"] Oct 30 17:15:00.248446 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.248362 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" Oct 30 17:15:00.256045 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.255999 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd"] Oct 30 17:15:00.274862 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.274840 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:15:00.284445 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.284412 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxvcb\" (UniqueName: \"kubernetes.io/projected/91c62170-6668-4436-b794-10170604b7d0-kube-api-access-qxvcb\") pod \"maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57\" (UID: \"91c62170-6668-4436-b794-10170604b7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" Oct 30 17:15:00.284643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.284484 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjnht\" (UniqueName: \"kubernetes.io/projected/c9267b55-cd1d-4d80-8c38-81fbdf673d64-kube-api-access-rjnht\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t\" (UID: \"c9267b55-cd1d-4d80-8c38-81fbdf673d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" Oct 30 17:15:00.284643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.284522 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9267b55-cd1d-4d80-8c38-81fbdf673d64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t\" (UID: \"c9267b55-cd1d-4d80-8c38-81fbdf673d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" Oct 30 17:15:00.284643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.284624 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvbwk\" (UniqueName: \"kubernetes.io/projected/510145d1-4707-4356-80de-09cf7da6540f-kube-api-access-jvbwk\") pod \"maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd\" (UID: \"510145d1-4707-4356-80de-09cf7da6540f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" Oct 30 17:15:00.284858 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.284728 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510145d1-4707-4356-80de-09cf7da6540f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd\" (UID: \"510145d1-4707-4356-80de-09cf7da6540f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" Oct 30 17:15:00.284858 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.284773 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c62170-6668-4436-b794-10170604b7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57\" (UID: \"91c62170-6668-4436-b794-10170604b7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" Oct 30 17:15:00.284858 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.284805 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htnt4\" (UniqueName: \"kubernetes.io/projected/75cea706-1357-4bf0-a987-6323bbf35317-kube-api-access-htnt4\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92\" (UID: \"75cea706-1357-4bf0-a987-6323bbf35317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" Oct 30 17:15:00.285036 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.284859 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cea706-1357-4bf0-a987-6323bbf35317-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92\" (UID: \"75cea706-1357-4bf0-a987-6323bbf35317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" Oct 30 17:15:00.287896 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.287875 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57"] Oct 30 17:15:00.386287 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.386253 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjnht\" (UniqueName: \"kubernetes.io/projected/c9267b55-cd1d-4d80-8c38-81fbdf673d64-kube-api-access-rjnht\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t\" (UID: \"c9267b55-cd1d-4d80-8c38-81fbdf673d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" Oct 30 17:15:00.386287 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.386292 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9267b55-cd1d-4d80-8c38-81fbdf673d64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t\" (UID: \"c9267b55-cd1d-4d80-8c38-81fbdf673d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" Oct 30 17:15:00.386540 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.386357 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvbwk\" (UniqueName: \"kubernetes.io/projected/510145d1-4707-4356-80de-09cf7da6540f-kube-api-access-jvbwk\") pod \"maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd\" (UID: \"510145d1-4707-4356-80de-09cf7da6540f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" Oct 30 17:15:00.386540 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.386430 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510145d1-4707-4356-80de-09cf7da6540f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd\" (UID: \"510145d1-4707-4356-80de-09cf7da6540f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" Oct 30 17:15:00.386540 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.386474 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c62170-6668-4436-b794-10170604b7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57\" (UID: \"91c62170-6668-4436-b794-10170604b7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" Oct 30 17:15:00.386540 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.386508 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htnt4\" (UniqueName: \"kubernetes.io/projected/75cea706-1357-4bf0-a987-6323bbf35317-kube-api-access-htnt4\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92\" (UID: \"75cea706-1357-4bf0-a987-6323bbf35317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" Oct 30 17:15:00.386800 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.386560 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cea706-1357-4bf0-a987-6323bbf35317-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92\" (UID: \"75cea706-1357-4bf0-a987-6323bbf35317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" Oct 30 17:15:00.386858 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.386791 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxvcb\" (UniqueName: \"kubernetes.io/projected/91c62170-6668-4436-b794-10170604b7d0-kube-api-access-qxvcb\") pod \"maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57\" (UID: \"91c62170-6668-4436-b794-10170604b7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" Oct 30 17:15:00.387246 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.387223 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510145d1-4707-4356-80de-09cf7da6540f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd\" (UID: \"510145d1-4707-4356-80de-09cf7da6540f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" Oct 30 17:15:00.387362 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.387253 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9267b55-cd1d-4d80-8c38-81fbdf673d64-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t\" (UID: \"c9267b55-cd1d-4d80-8c38-81fbdf673d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" Oct 30 17:15:00.387362 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.387259 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cea706-1357-4bf0-a987-6323bbf35317-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92\" (UID: \"75cea706-1357-4bf0-a987-6323bbf35317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" Oct 30 17:15:00.387362 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.387261 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c62170-6668-4436-b794-10170604b7d0-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57\" (UID: \"91c62170-6668-4436-b794-10170604b7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" Oct 30 17:15:00.411517 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.411484 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjnht\" (UniqueName: \"kubernetes.io/projected/c9267b55-cd1d-4d80-8c38-81fbdf673d64-kube-api-access-rjnht\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t\" (UID: \"c9267b55-cd1d-4d80-8c38-81fbdf673d64\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" Oct 30 17:15:00.411690 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.411516 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvbwk\" (UniqueName: \"kubernetes.io/projected/510145d1-4707-4356-80de-09cf7da6540f-kube-api-access-jvbwk\") pod \"maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd\" (UID: \"510145d1-4707-4356-80de-09cf7da6540f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" Oct 30 17:15:00.411690 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.411489 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htnt4\" (UniqueName: \"kubernetes.io/projected/75cea706-1357-4bf0-a987-6323bbf35317-kube-api-access-htnt4\") pod \"maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92\" (UID: \"75cea706-1357-4bf0-a987-6323bbf35317\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" Oct 30 17:15:00.411690 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.411574 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxvcb\" (UniqueName: \"kubernetes.io/projected/91c62170-6668-4436-b794-10170604b7d0-kube-api-access-qxvcb\") pod \"maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57\" (UID: \"91c62170-6668-4436-b794-10170604b7d0\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" Oct 30 17:15:00.515682 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.515644 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" Oct 30 17:15:00.554074 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.554035 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" Oct 30 17:15:00.572961 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.572929 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" Oct 30 17:15:00.581663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.581594 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" Oct 30 17:15:00.831306 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:15:00.831277 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9267b55_cd1d_4d80_8c38_81fbdf673d64.slice/crio-a87771da71683a6eee49193d1b20db89ebd96bbebee04522c6c131fd894415e1 WatchSource:0}: Error finding container a87771da71683a6eee49193d1b20db89ebd96bbebee04522c6c131fd894415e1: Status 404 returned error can't find the container with id a87771da71683a6eee49193d1b20db89ebd96bbebee04522c6c131fd894415e1 Oct 30 17:15:00.832283 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.832262 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t"] Oct 30 17:15:00.905813 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:15:00.905785 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75cea706_1357_4bf0_a987_6323bbf35317.slice/crio-31fd118c74a0d50a53eef8976df1a129a4b04674a04d45160a682e1ad4a5f69f WatchSource:0}: Error finding container 31fd118c74a0d50a53eef8976df1a129a4b04674a04d45160a682e1ad4a5f69f: Status 404 returned error can't find the container with id 31fd118c74a0d50a53eef8976df1a129a4b04674a04d45160a682e1ad4a5f69f Oct 30 17:15:00.906855 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.906836 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92"] Oct 30 17:15:00.953561 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.953340 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd"] Oct 30 17:15:00.958763 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:00.958738 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57"] Oct 30 17:15:01.012412 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:15:01.012380 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510145d1_4707_4356_80de_09cf7da6540f.slice/crio-07ef145b4ca7e1edef8b9bcaa7366c7627502682af244d4948023dc1c0c3e029 WatchSource:0}: Error finding container 07ef145b4ca7e1edef8b9bcaa7366c7627502682af244d4948023dc1c0c3e029: Status 404 returned error can't find the container with id 07ef145b4ca7e1edef8b9bcaa7366c7627502682af244d4948023dc1c0c3e029 Oct 30 17:15:01.013418 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:15:01.013382 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91c62170_6668_4436_b794_10170604b7d0.slice/crio-59f5e8f98daa797f5949a74decfa9595e2b914471478526341dcd97513672e1f WatchSource:0}: Error finding container 59f5e8f98daa797f5949a74decfa9595e2b914471478526341dcd97513672e1f: Status 404 returned error can't find the container with id 59f5e8f98daa797f5949a74decfa9595e2b914471478526341dcd97513672e1f Oct 30 17:15:01.340385 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:01.340349 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" event={"ID":"510145d1-4707-4356-80de-09cf7da6540f","Type":"ContainerStarted","Data":"07ef145b4ca7e1edef8b9bcaa7366c7627502682af244d4948023dc1c0c3e029"} Oct 30 17:15:01.342143 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:01.342115 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" event={"ID":"c9267b55-cd1d-4d80-8c38-81fbdf673d64","Type":"ContainerStarted","Data":"c679e459eebee393a52f84895d84bb7632f321c606cddf78fd4b47783cb278ae"} Oct 30 17:15:01.342293 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:01.342172 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" event={"ID":"c9267b55-cd1d-4d80-8c38-81fbdf673d64","Type":"ContainerStarted","Data":"a87771da71683a6eee49193d1b20db89ebd96bbebee04522c6c131fd894415e1"} Oct 30 17:15:01.343345 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:01.343320 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" event={"ID":"91c62170-6668-4436-b794-10170604b7d0","Type":"ContainerStarted","Data":"59f5e8f98daa797f5949a74decfa9595e2b914471478526341dcd97513672e1f"} Oct 30 17:15:01.344555 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:01.344522 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" event={"ID":"75cea706-1357-4bf0-a987-6323bbf35317","Type":"ContainerStarted","Data":"31fd118c74a0d50a53eef8976df1a129a4b04674a04d45160a682e1ad4a5f69f"} Oct 30 17:15:01.394229 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:01.394151 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" podStartSLOduration=1.394129001 podStartE2EDuration="1.394129001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:01.391759406 +0000 UTC m=+3210.542040397" watchObservedRunningTime="2026-10-30 17:15:01.394129001 +0000 UTC m=+3210.544409993" Oct 30 17:15:01.952928 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:01.952897 6400 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod510145d1_4707_4356_80de_09cf7da6540f.slice/crio-21fce7cdafba3e59742353cf1f5f9b85cd5247e7153598a334e4ddad753aba5e.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:15:02.352895 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.352848 6400 generic.go:356] "Generic (PLEG): container finished" podID="91c62170-6668-4436-b794-10170604b7d0" containerID="d3686eb9be223aebcc49fb0f97c2ccb19b9dad97fc3cdfd4fc2d622f039df406" exitCode=60 Oct 30 17:15:02.353358 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.352933 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" event={"ID":"91c62170-6668-4436-b794-10170604b7d0","Type":"ContainerDied","Data":"d3686eb9be223aebcc49fb0f97c2ccb19b9dad97fc3cdfd4fc2d622f039df406"} Oct 30 17:15:02.353358 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.353281 6400 scope.go:122] "RemoveContainer" containerID="d3686eb9be223aebcc49fb0f97c2ccb19b9dad97fc3cdfd4fc2d622f039df406" Oct 30 17:15:02.354829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.354807 6400 generic.go:356] "Generic (PLEG): container finished" podID="75cea706-1357-4bf0-a987-6323bbf35317" containerID="874f5ea263002d74c4e3002a9df4abb7bbf7b02c35a33571417bac34ae7467bc" exitCode=60 Oct 30 17:15:02.354949 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.354888 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" event={"ID":"75cea706-1357-4bf0-a987-6323bbf35317","Type":"ContainerDied","Data":"874f5ea263002d74c4e3002a9df4abb7bbf7b02c35a33571417bac34ae7467bc"} Oct 30 17:15:02.355161 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.355140 6400 scope.go:122] "RemoveContainer" containerID="874f5ea263002d74c4e3002a9df4abb7bbf7b02c35a33571417bac34ae7467bc" Oct 30 17:15:02.356676 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.356656 6400 generic.go:356] "Generic (PLEG): container finished" podID="510145d1-4707-4356-80de-09cf7da6540f" containerID="21fce7cdafba3e59742353cf1f5f9b85cd5247e7153598a334e4ddad753aba5e" exitCode=60 Oct 30 17:15:02.356865 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.356736 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" event={"ID":"510145d1-4707-4356-80de-09cf7da6540f","Type":"ContainerDied","Data":"21fce7cdafba3e59742353cf1f5f9b85cd5247e7153598a334e4ddad753aba5e"} Oct 30 17:15:02.357230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.357004 6400 scope.go:122] "RemoveContainer" containerID="21fce7cdafba3e59742353cf1f5f9b85cd5247e7153598a334e4ddad753aba5e" Oct 30 17:15:02.359196 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.359174 6400 generic.go:356] "Generic (PLEG): container finished" podID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerID="c679e459eebee393a52f84895d84bb7632f321c606cddf78fd4b47783cb278ae" exitCode=60 Oct 30 17:15:02.359285 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.359257 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" event={"ID":"c9267b55-cd1d-4d80-8c38-81fbdf673d64","Type":"ContainerDied","Data":"c679e459eebee393a52f84895d84bb7632f321c606cddf78fd4b47783cb278ae"} Oct 30 17:15:02.359449 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:02.359431 6400 scope.go:122] "RemoveContainer" containerID="c679e459eebee393a52f84895d84bb7632f321c606cddf78fd4b47783cb278ae" Oct 30 17:15:03.368201 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:03.368168 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" event={"ID":"91c62170-6668-4436-b794-10170604b7d0","Type":"ContainerStarted","Data":"06caa9d4b59c5fb5e93725c76a25920a462e9bbcf360460bdd58db0f8d7d60c7"} Oct 30 17:15:03.370099 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:03.370072 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" event={"ID":"75cea706-1357-4bf0-a987-6323bbf35317","Type":"ContainerStarted","Data":"716ea7f0474d22eaeb2e9d379077eb59e2e2e5adfb6c1cbca856712cc6fe3f03"} Oct 30 17:15:03.372236 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:03.372189 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" event={"ID":"510145d1-4707-4356-80de-09cf7da6540f","Type":"ContainerStarted","Data":"5e795a5a9779c3d36ac806232460109700fe2e5afab4facbdf7dc868c2160ea1"} Oct 30 17:15:03.374634 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:03.374587 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" event={"ID":"c9267b55-cd1d-4d80-8c38-81fbdf673d64","Type":"ContainerStarted","Data":"360bb60786a2f03657095e668636f9267663c4369c586444f9d8836f06666292"} Oct 30 17:15:03.420502 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:03.420406 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" podStartSLOduration=3.420381677 podStartE2EDuration="3.420381677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.416501781 +0000 UTC m=+3212.566782768" watchObservedRunningTime="2026-10-30 17:15:03.420381677 +0000 UTC m=+3212.570662665" Oct 30 17:15:03.503171 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:03.503112 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" podStartSLOduration=3.50309807 podStartE2EDuration="3.50309807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.500472448 +0000 UTC m=+3212.650753436" watchObservedRunningTime="2026-10-30 17:15:03.50309807 +0000 UTC m=+3212.653379061" Oct 30 17:15:03.547848 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:03.547785 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" podStartSLOduration=3.547764917 podStartE2EDuration="3.547764917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:03.544897566 +0000 UTC m=+3212.695178555" watchObservedRunningTime="2026-10-30 17:15:03.547764917 +0000 UTC m=+3212.698045905" Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.390832 6400 generic.go:356] "Generic (PLEG): container finished" podID="510145d1-4707-4356-80de-09cf7da6540f" containerID="5e795a5a9779c3d36ac806232460109700fe2e5afab4facbdf7dc868c2160ea1" exitCode=60 Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.390925 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" event={"ID":"510145d1-4707-4356-80de-09cf7da6540f","Type":"ContainerDied","Data":"5e795a5a9779c3d36ac806232460109700fe2e5afab4facbdf7dc868c2160ea1"} Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.390979 6400 scope.go:122] "RemoveContainer" containerID="21fce7cdafba3e59742353cf1f5f9b85cd5247e7153598a334e4ddad753aba5e" Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.391550 6400 scope.go:122] "RemoveContainer" containerID="5e795a5a9779c3d36ac806232460109700fe2e5afab4facbdf7dc868c2160ea1" Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:04.392024 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd_odh-ai-gateway-infra(510145d1-4707-4356-80de-09cf7da6540f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" podUID="510145d1-4707-4356-80de-09cf7da6540f" Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.397692 6400 generic.go:356] "Generic (PLEG): container finished" podID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerID="360bb60786a2f03657095e668636f9267663c4369c586444f9d8836f06666292" exitCode=60 Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.397937 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" event={"ID":"c9267b55-cd1d-4d80-8c38-81fbdf673d64","Type":"ContainerDied","Data":"360bb60786a2f03657095e668636f9267663c4369c586444f9d8836f06666292"} Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.398390 6400 scope.go:122] "RemoveContainer" containerID="360bb60786a2f03657095e668636f9267663c4369c586444f9d8836f06666292" Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:04.398799 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t_odh-ai-gateway-infra(c9267b55-cd1d-4d80-8c38-81fbdf673d64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.401775 6400 generic.go:356] "Generic (PLEG): container finished" podID="91c62170-6668-4436-b794-10170604b7d0" containerID="06caa9d4b59c5fb5e93725c76a25920a462e9bbcf360460bdd58db0f8d7d60c7" exitCode=60 Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.401873 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" event={"ID":"91c62170-6668-4436-b794-10170604b7d0","Type":"ContainerDied","Data":"06caa9d4b59c5fb5e93725c76a25920a462e9bbcf360460bdd58db0f8d7d60c7"} Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.402259 6400 scope.go:122] "RemoveContainer" containerID="06caa9d4b59c5fb5e93725c76a25920a462e9bbcf360460bdd58db0f8d7d60c7" Oct 30 17:15:04.403636 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:04.402585 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57_odh-ai-gateway-infra(91c62170-6668-4436-b794-10170604b7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" podUID="91c62170-6668-4436-b794-10170604b7d0" Oct 30 17:15:04.408025 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.407996 6400 generic.go:356] "Generic (PLEG): container finished" podID="75cea706-1357-4bf0-a987-6323bbf35317" containerID="716ea7f0474d22eaeb2e9d379077eb59e2e2e5adfb6c1cbca856712cc6fe3f03" exitCode=60 Oct 30 17:15:04.414488 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.414463 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" event={"ID":"75cea706-1357-4bf0-a987-6323bbf35317","Type":"ContainerDied","Data":"716ea7f0474d22eaeb2e9d379077eb59e2e2e5adfb6c1cbca856712cc6fe3f03"} Oct 30 17:15:04.421834 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.419556 6400 scope.go:122] "RemoveContainer" containerID="716ea7f0474d22eaeb2e9d379077eb59e2e2e5adfb6c1cbca856712cc6fe3f03" Oct 30 17:15:04.424038 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:04.422366 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92_odh-ai-gateway-infra(75cea706-1357-4bf0-a987-6323bbf35317)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" podUID="75cea706-1357-4bf0-a987-6323bbf35317" Oct 30 17:15:04.432904 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.432820 6400 scope.go:122] "RemoveContainer" containerID="c679e459eebee393a52f84895d84bb7632f321c606cddf78fd4b47783cb278ae" Oct 30 17:15:04.456806 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.456773 6400 scope.go:122] "RemoveContainer" containerID="d3686eb9be223aebcc49fb0f97c2ccb19b9dad97fc3cdfd4fc2d622f039df406" Oct 30 17:15:04.472410 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:04.472390 6400 scope.go:122] "RemoveContainer" containerID="874f5ea263002d74c4e3002a9df4abb7bbf7b02c35a33571417bac34ae7467bc" Oct 30 17:15:05.424039 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.424001 6400 scope.go:122] "RemoveContainer" containerID="06caa9d4b59c5fb5e93725c76a25920a462e9bbcf360460bdd58db0f8d7d60c7" Oct 30 17:15:05.424556 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:05.424346 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57_odh-ai-gateway-infra(91c62170-6668-4436-b794-10170604b7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" podUID="91c62170-6668-4436-b794-10170604b7d0" Oct 30 17:15:05.425749 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.425727 6400 scope.go:122] "RemoveContainer" containerID="716ea7f0474d22eaeb2e9d379077eb59e2e2e5adfb6c1cbca856712cc6fe3f03" Oct 30 17:15:05.426061 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:05.426035 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92_odh-ai-gateway-infra(75cea706-1357-4bf0-a987-6323bbf35317)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" podUID="75cea706-1357-4bf0-a987-6323bbf35317" Oct 30 17:15:05.427410 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.427390 6400 scope.go:122] "RemoveContainer" containerID="5e795a5a9779c3d36ac806232460109700fe2e5afab4facbdf7dc868c2160ea1" Oct 30 17:15:05.427741 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:05.427718 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd_odh-ai-gateway-infra(510145d1-4707-4356-80de-09cf7da6540f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" podUID="510145d1-4707-4356-80de-09cf7da6540f" Oct 30 17:15:05.428944 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.428929 6400 scope.go:122] "RemoveContainer" containerID="360bb60786a2f03657095e668636f9267663c4369c586444f9d8836f06666292" Oct 30 17:15:05.429167 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:05.429149 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t_odh-ai-gateway-infra(c9267b55-cd1d-4d80-8c38-81fbdf673d64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" Oct 30 17:15:05.787550 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.787508 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j"] Oct 30 17:15:05.798355 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.798324 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:05.809677 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.809287 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j"] Oct 30 17:15:05.859075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.859043 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:05.871807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.871777 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d001992-9272-4b14-a48d-8468520e5975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:05.871963 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.871938 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d001992-9272-4b14-a48d-8468520e5975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:05.872038 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.871983 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4q6mf\" (UniqueName: \"kubernetes.io/projected/4d001992-9272-4b14-a48d-8468520e5975-kube-api-access-4q6mf\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:05.973239 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.973206 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d001992-9272-4b14-a48d-8468520e5975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:05.973416 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.973338 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d001992-9272-4b14-a48d-8468520e5975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:05.973416 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.973392 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4q6mf\" (UniqueName: \"kubernetes.io/projected/4d001992-9272-4b14-a48d-8468520e5975-kube-api-access-4q6mf\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:05.973948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.973930 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d001992-9272-4b14-a48d-8468520e5975-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:05.975925 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:05.975905 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d001992-9272-4b14-a48d-8468520e5975-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:06.000943 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:06.000916 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4q6mf\" (UniqueName: \"kubernetes.io/projected/4d001992-9272-4b14-a48d-8468520e5975-kube-api-access-4q6mf\") pod \"maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:06.122335 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:06.122240 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:06.433443 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:15:06.433414 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d001992_9272_4b14_a48d_8468520e5975.slice/crio-2c7f11bd7ec30045bb4e2c544c4518cf13bb97ae15c65af58a469fc50b3a4753 WatchSource:0}: Error finding container 2c7f11bd7ec30045bb4e2c544c4518cf13bb97ae15c65af58a469fc50b3a4753: Status 404 returned error can't find the container with id 2c7f11bd7ec30045bb4e2c544c4518cf13bb97ae15c65af58a469fc50b3a4753 Oct 30 17:15:06.433931 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:06.433909 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j"] Oct 30 17:15:07.445763 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:07.445721 6400 generic.go:356] "Generic (PLEG): container finished" podID="4d001992-9272-4b14-a48d-8468520e5975" containerID="dd8c23379d6f83137bd0b30ace1f3d9862d6928516550f97b798e0c89cd060ca" exitCode=0 Oct 30 17:15:07.446170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:07.445787 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" event={"ID":"4d001992-9272-4b14-a48d-8468520e5975","Type":"ContainerDied","Data":"dd8c23379d6f83137bd0b30ace1f3d9862d6928516550f97b798e0c89cd060ca"} Oct 30 17:15:07.446170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:07.445855 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" event={"ID":"4d001992-9272-4b14-a48d-8468520e5975","Type":"ContainerStarted","Data":"2c7f11bd7ec30045bb4e2c544c4518cf13bb97ae15c65af58a469fc50b3a4753"} Oct 30 17:15:08.661105 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:08.661079 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:08.819794 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:08.819757 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d001992-9272-4b14-a48d-8468520e5975-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d001992-9272-4b14-a48d-8468520e5975-maas-api-xks-ca\") pod \"4d001992-9272-4b14-a48d-8468520e5975\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " Oct 30 17:15:08.819967 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:08.819854 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d001992-9272-4b14-a48d-8468520e5975-kube-api-access-4q6mf\" (UniqueName: \"kubernetes.io/projected/4d001992-9272-4b14-a48d-8468520e5975-kube-api-access-4q6mf\") pod \"4d001992-9272-4b14-a48d-8468520e5975\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " Oct 30 17:15:08.820032 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:08.820018 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d001992-9272-4b14-a48d-8468520e5975-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d001992-9272-4b14-a48d-8468520e5975-maas-api-openshift-ca\") pod \"4d001992-9272-4b14-a48d-8468520e5975\" (UID: \"4d001992-9272-4b14-a48d-8468520e5975\") " Oct 30 17:15:08.820479 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:08.820454 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d001992-9272-4b14-a48d-8468520e5975-maas-api-openshift-ca" pod "4d001992-9272-4b14-a48d-8468520e5975" (UID: "4d001992-9272-4b14-a48d-8468520e5975"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:08.822404 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:08.822376 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d001992-9272-4b14-a48d-8468520e5975-maas-api-xks-ca" pod "4d001992-9272-4b14-a48d-8468520e5975" (UID: "4d001992-9272-4b14-a48d-8468520e5975"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:08.822547 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:08.822499 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d001992-9272-4b14-a48d-8468520e5975-kube-api-access-4q6mf" pod "4d001992-9272-4b14-a48d-8468520e5975" (UID: "4d001992-9272-4b14-a48d-8468520e5975"). InnerVolumeSpecName "kube-api-access-4q6mf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:08.921841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:08.921801 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d001992-9272-4b14-a48d-8468520e5975-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:08.921841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:08.921836 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4q6mf\" (UniqueName: \"kubernetes.io/projected/4d001992-9272-4b14-a48d-8468520e5975-kube-api-access-4q6mf\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:08.921841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:08.921853 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d001992-9272-4b14-a48d-8468520e5975-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:09.462842 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:09.462812 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" Oct 30 17:15:09.463039 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:09.462845 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j" event={"ID":"4d001992-9272-4b14-a48d-8468520e5975","Type":"ContainerDied","Data":"2c7f11bd7ec30045bb4e2c544c4518cf13bb97ae15c65af58a469fc50b3a4753"} Oct 30 17:15:09.463039 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:09.462897 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c7f11bd7ec30045bb4e2c544c4518cf13bb97ae15c65af58a469fc50b3a4753" Oct 30 17:15:09.958225 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:09.958191 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2"] Oct 30 17:15:09.958764 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:09.958454 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" containerID="cri-o://6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d" gracePeriod=30 Oct 30 17:15:12.369555 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:12.369520 6400 patch_prober.go:29] interesting pod/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:15:12.369964 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:12.369574 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:15:14.960718 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:14.960683 6400 scope.go:122] "RemoveContainer" containerID="360bb60786a2f03657095e668636f9267663c4369c586444f9d8836f06666292" Oct 30 17:15:15.218406 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.218349 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 17:15:15.302527 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.302490 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-podinfo\") pod \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " Oct 30 17:15:15.302760 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.302544 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-credential-socket\") pod \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " Oct 30 17:15:15.302760 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.302574 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istiod-ca-cert\") pod \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " Oct 30 17:15:15.302760 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.302686 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-kube-api-access-lp9ms\" (UniqueName: \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-kube-api-access-lp9ms\") pod \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " Oct 30 17:15:15.302954 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.302827 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-certs\") pod \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " Oct 30 17:15:15.302954 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.302902 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-credential-socket" pod "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" (UID: "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:15.303063 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.302962 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-token\" (UniqueName: \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-token\") pod \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " Oct 30 17:15:15.303063 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303025 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-socket\") pod \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " Oct 30 17:15:15.303175 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303063 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-envoy\") pod \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " Oct 30 17:15:15.303175 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303103 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-data\") pod \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\" (UID: \"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89\") " Oct 30 17:15:15.303536 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303097 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istiod-ca-cert" pod "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" (UID: "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:15.303536 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303235 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-certs" pod "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" (UID: "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:15.303536 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303385 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-socket" pod "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" (UID: "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:15.303815 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303776 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:15.303960 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303878 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:15.304043 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303779 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-data" pod "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" (UID: "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:15.304043 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303973 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:15.304043 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.303989 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:15.305889 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.305863 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-podinfo" pod "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" (UID: "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:15.305889 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.305875 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-token" pod "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" (UID: "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:15.306799 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.306779 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-kube-api-access-lp9ms" pod "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" (UID: "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89"). InnerVolumeSpecName "kube-api-access-lp9ms". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:15.307053 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.307030 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-envoy" pod "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" (UID: "61744ea0-6fdc-4a3c-bcba-f93f6aa69d89"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:15.405154 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.405116 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp9ms\" (UniqueName: \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-kube-api-access-lp9ms\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:15.405154 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.405149 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:15.405154 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.405162 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:15.405397 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.405173 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:15.405397 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.405183 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:15.507316 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.507280 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" event={"ID":"c9267b55-cd1d-4d80-8c38-81fbdf673d64","Type":"ContainerStarted","Data":"c6074ab069ce00ff4c1a3c461b732d98b409b9959084ec2dc68e22728f7c04f9"} Oct 30 17:15:15.508874 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.508847 6400 generic.go:356] "Generic (PLEG): container finished" podID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerID="6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d" exitCode=0 Oct 30 17:15:15.509008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.508895 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" Oct 30 17:15:15.509008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.508926 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" event={"ID":"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89","Type":"ContainerDied","Data":"6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d"} Oct 30 17:15:15.509008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.508980 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2" event={"ID":"61744ea0-6fdc-4a3c-bcba-f93f6aa69d89","Type":"ContainerDied","Data":"09ddbd4c1e3a58304ab2b1ed5e5812fece24cc726cf034ead9ad2fee45589182"} Oct 30 17:15:15.509008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.509005 6400 scope.go:122] "RemoveContainer" containerID="6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d" Oct 30 17:15:15.520025 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.520008 6400 scope.go:122] "RemoveContainer" containerID="6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d" Oct 30 17:15:15.520288 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:15.520271 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d\": container with ID starting with 6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d not found: ID does not exist" containerID="6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d" Oct 30 17:15:15.520374 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.520295 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d"} err="failed to get container status \"6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d\": rpc error: code = NotFound desc = could not find container \"6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d\": container with ID starting with 6ff773c50628882430644007b1e0d10013c937986af24d6518287d2432e3389d not found: ID does not exist" Oct 30 17:15:15.594583 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.594544 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2"] Oct 30 17:15:15.597986 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.597956 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-3e2f96-openshift-default-84865466dc-nmhx2"] Oct 30 17:15:15.960431 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:15.960353 6400 scope.go:122] "RemoveContainer" containerID="716ea7f0474d22eaeb2e9d379077eb59e2e2e5adfb6c1cbca856712cc6fe3f03" Oct 30 17:15:16.518224 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:16.518182 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" event={"ID":"75cea706-1357-4bf0-a987-6323bbf35317","Type":"ContainerStarted","Data":"545cf66f68220880d6a610ed687560632016dc7a895b5e18e4449b1f3a6b9afc"} Oct 30 17:15:16.520191 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:16.520164 6400 generic.go:356] "Generic (PLEG): container finished" podID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerID="c6074ab069ce00ff4c1a3c461b732d98b409b9959084ec2dc68e22728f7c04f9" exitCode=60 Oct 30 17:15:16.520298 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:16.520226 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" event={"ID":"c9267b55-cd1d-4d80-8c38-81fbdf673d64","Type":"ContainerDied","Data":"c6074ab069ce00ff4c1a3c461b732d98b409b9959084ec2dc68e22728f7c04f9"} Oct 30 17:15:16.520298 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:16.520279 6400 scope.go:122] "RemoveContainer" containerID="360bb60786a2f03657095e668636f9267663c4369c586444f9d8836f06666292" Oct 30 17:15:16.520563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:16.520547 6400 scope.go:122] "RemoveContainer" containerID="c6074ab069ce00ff4c1a3c461b732d98b409b9959084ec2dc68e22728f7c04f9" Oct 30 17:15:16.520815 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:16.520795 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t_odh-ai-gateway-infra(c9267b55-cd1d-4d80-8c38-81fbdf673d64)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" Oct 30 17:15:16.560997 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:16.560969 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t"] Oct 30 17:15:16.981121 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:16.981087 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" path="/var/lib/kubelet/pods/61744ea0-6fdc-4a3c-bcba-f93f6aa69d89/volumes" Oct 30 17:15:17.400370 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.400334 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z"] Oct 30 17:15:17.400970 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.400955 6400 cpu_manager.go:414] "containerMap: removing container" podUID="4d001992-9272-4b14-a48d-8468520e5975" containerName="revoke-keys" Oct 30 17:15:17.401042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.400971 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d001992-9272-4b14-a48d-8468520e5975" containerName="revoke-keys" Oct 30 17:15:17.401042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.400991 6400 cpu_manager.go:414] "containerMap: removing container" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" Oct 30 17:15:17.401042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.400997 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" Oct 30 17:15:17.401146 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.401096 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d001992-9272-4b14-a48d-8468520e5975" containerName="revoke-keys" Oct 30 17:15:17.401146 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.401108 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="61744ea0-6fdc-4a3c-bcba-f93f6aa69d89" containerName="istio-proxy" Oct 30 17:15:17.406568 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.406376 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.425354 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.425322 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z"] Oct 30 17:15:17.430688 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.430658 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnqj2\" (UniqueName: \"kubernetes.io/projected/e32f582c-5e90-4129-aa85-cdf31c99d0ac-kube-api-access-nnqj2\") pod \"maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.430828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.430695 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.430922 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.430822 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.498780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.498746 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:15:17.529060 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.529022 6400 generic.go:356] "Generic (PLEG): container finished" podID="75cea706-1357-4bf0-a987-6323bbf35317" containerID="545cf66f68220880d6a610ed687560632016dc7a895b5e18e4449b1f3a6b9afc" exitCode=60 Oct 30 17:15:17.529525 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.529085 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" event={"ID":"75cea706-1357-4bf0-a987-6323bbf35317","Type":"ContainerDied","Data":"545cf66f68220880d6a610ed687560632016dc7a895b5e18e4449b1f3a6b9afc"} Oct 30 17:15:17.529525 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.529167 6400 scope.go:122] "RemoveContainer" containerID="716ea7f0474d22eaeb2e9d379077eb59e2e2e5adfb6c1cbca856712cc6fe3f03" Oct 30 17:15:17.529695 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.529583 6400 scope.go:122] "RemoveContainer" containerID="545cf66f68220880d6a610ed687560632016dc7a895b5e18e4449b1f3a6b9afc" Oct 30 17:15:17.529950 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:17.529931 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92_odh-ai-gateway-infra(75cea706-1357-4bf0-a987-6323bbf35317)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" podUID="75cea706-1357-4bf0-a987-6323bbf35317" Oct 30 17:15:17.531783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.531762 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnqj2\" (UniqueName: \"kubernetes.io/projected/e32f582c-5e90-4129-aa85-cdf31c99d0ac-kube-api-access-nnqj2\") pod \"maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.531910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.531794 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.531910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.531865 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.532721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.532520 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.534666 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.534642 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.558739 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.558708 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnqj2\" (UniqueName: \"kubernetes.io/projected/e32f582c-5e90-4129-aa85-cdf31c99d0ac-kube-api-access-nnqj2\") pod \"maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.575035 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.575000 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92"] Oct 30 17:15:17.725763 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.725735 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:17.731918 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.731889 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" Oct 30 17:15:17.836382 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.836348 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9267b55-cd1d-4d80-8c38-81fbdf673d64-kube-api-access-rjnht\" (UniqueName: \"kubernetes.io/projected/c9267b55-cd1d-4d80-8c38-81fbdf673d64-kube-api-access-rjnht\") pod \"c9267b55-cd1d-4d80-8c38-81fbdf673d64\" (UID: \"c9267b55-cd1d-4d80-8c38-81fbdf673d64\") " Oct 30 17:15:17.836563 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.836481 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9267b55-cd1d-4d80-8c38-81fbdf673d64-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9267b55-cd1d-4d80-8c38-81fbdf673d64-maas-api-service-ca\") pod \"c9267b55-cd1d-4d80-8c38-81fbdf673d64\" (UID: \"c9267b55-cd1d-4d80-8c38-81fbdf673d64\") " Oct 30 17:15:17.836962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.836939 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9267b55-cd1d-4d80-8c38-81fbdf673d64-maas-api-service-ca" pod "c9267b55-cd1d-4d80-8c38-81fbdf673d64" (UID: "c9267b55-cd1d-4d80-8c38-81fbdf673d64"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:17.837208 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.837181 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c9267b55-cd1d-4d80-8c38-81fbdf673d64-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.839269 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.839237 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9267b55-cd1d-4d80-8c38-81fbdf673d64-kube-api-access-rjnht" pod "c9267b55-cd1d-4d80-8c38-81fbdf673d64" (UID: "c9267b55-cd1d-4d80-8c38-81fbdf673d64"). InnerVolumeSpecName "kube-api-access-rjnht". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:17.938952 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.938914 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjnht\" (UniqueName: \"kubernetes.io/projected/c9267b55-cd1d-4d80-8c38-81fbdf673d64-kube-api-access-rjnht\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:17.960846 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.960759 6400 scope.go:122] "RemoveContainer" containerID="06caa9d4b59c5fb5e93725c76a25920a462e9bbcf360460bdd58db0f8d7d60c7" Oct 30 17:15:17.961027 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:17.960935 6400 scope.go:122] "RemoveContainer" containerID="5e795a5a9779c3d36ac806232460109700fe2e5afab4facbdf7dc868c2160ea1" Oct 30 17:15:18.038394 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:15:18.038363 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode32f582c_5e90_4129_aa85_cdf31c99d0ac.slice/crio-6239d82ea3441b941d92c301adc818129697f5a8f0219b225d094d796ab48d66 WatchSource:0}: Error finding container 6239d82ea3441b941d92c301adc818129697f5a8f0219b225d094d796ab48d66: Status 404 returned error can't find the container with id 6239d82ea3441b941d92c301adc818129697f5a8f0219b225d094d796ab48d66 Oct 30 17:15:18.038786 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.038763 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z"] Oct 30 17:15:18.544157 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.544120 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" event={"ID":"91c62170-6668-4436-b794-10170604b7d0","Type":"ContainerStarted","Data":"e41235119f3e022efa87a80b3fb49d1367d5c151df3e3f1040db33492380fd44"} Oct 30 17:15:18.550024 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.549987 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" event={"ID":"510145d1-4707-4356-80de-09cf7da6540f","Type":"ContainerStarted","Data":"bded2cf2d0b2a7f178438bffd6b95ede9de51ccef6a76af11d753443cf69d2df"} Oct 30 17:15:18.552167 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.552137 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" Oct 30 17:15:18.552290 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.552195 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t" event={"ID":"c9267b55-cd1d-4d80-8c38-81fbdf673d64","Type":"ContainerDied","Data":"a87771da71683a6eee49193d1b20db89ebd96bbebee04522c6c131fd894415e1"} Oct 30 17:15:18.552363 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.552288 6400 scope.go:122] "RemoveContainer" containerID="c6074ab069ce00ff4c1a3c461b732d98b409b9959084ec2dc68e22728f7c04f9" Oct 30 17:15:18.554389 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.554363 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" event={"ID":"e32f582c-5e90-4129-aa85-cdf31c99d0ac","Type":"ContainerStarted","Data":"b58492d331aaf3304dbdbc0774b144fef2cb11da4f56f1959180b51d56b04b1f"} Oct 30 17:15:18.554504 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.554409 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" event={"ID":"e32f582c-5e90-4129-aa85-cdf31c99d0ac","Type":"ContainerStarted","Data":"6239d82ea3441b941d92c301adc818129697f5a8f0219b225d094d796ab48d66"} Oct 30 17:15:18.718534 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.718507 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" Oct 30 17:15:18.720210 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.720170 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" podStartSLOduration=1.7201574929999999 podStartE2EDuration="1.720157493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:15:18.718171831 +0000 UTC m=+3227.868452823" watchObservedRunningTime="2026-10-30 17:15:18.720157493 +0000 UTC m=+3227.870438483" Oct 30 17:15:18.748444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.748407 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75cea706-1357-4bf0-a987-6323bbf35317-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cea706-1357-4bf0-a987-6323bbf35317-maas-api-service-ca\") pod \"75cea706-1357-4bf0-a987-6323bbf35317\" (UID: \"75cea706-1357-4bf0-a987-6323bbf35317\") " Oct 30 17:15:18.748625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.748595 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75cea706-1357-4bf0-a987-6323bbf35317-kube-api-access-htnt4\" (UniqueName: \"kubernetes.io/projected/75cea706-1357-4bf0-a987-6323bbf35317-kube-api-access-htnt4\") pod \"75cea706-1357-4bf0-a987-6323bbf35317\" (UID: \"75cea706-1357-4bf0-a987-6323bbf35317\") " Oct 30 17:15:18.748807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.748781 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75cea706-1357-4bf0-a987-6323bbf35317-maas-api-service-ca" pod "75cea706-1357-4bf0-a987-6323bbf35317" (UID: "75cea706-1357-4bf0-a987-6323bbf35317"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:18.749088 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.749066 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75cea706-1357-4bf0-a987-6323bbf35317-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.751061 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.751037 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75cea706-1357-4bf0-a987-6323bbf35317-kube-api-access-htnt4" pod "75cea706-1357-4bf0-a987-6323bbf35317" (UID: "75cea706-1357-4bf0-a987-6323bbf35317"). InnerVolumeSpecName "kube-api-access-htnt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:18.757967 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.757945 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t"] Oct 30 17:15:18.760839 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.760819 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-661509-29889675-4dg5t"] Oct 30 17:15:18.850057 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.849984 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htnt4\" (UniqueName: \"kubernetes.io/projected/75cea706-1357-4bf0-a987-6323bbf35317-kube-api-access-htnt4\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:18.968137 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:18.968101 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" path="/var/lib/kubelet/pods/c9267b55-cd1d-4d80-8c38-81fbdf673d64/volumes" Oct 30 17:15:19.563466 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.563433 6400 generic.go:356] "Generic (PLEG): container finished" podID="91c62170-6668-4436-b794-10170604b7d0" containerID="e41235119f3e022efa87a80b3fb49d1367d5c151df3e3f1040db33492380fd44" exitCode=60 Oct 30 17:15:19.563911 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.563506 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" event={"ID":"91c62170-6668-4436-b794-10170604b7d0","Type":"ContainerDied","Data":"e41235119f3e022efa87a80b3fb49d1367d5c151df3e3f1040db33492380fd44"} Oct 30 17:15:19.563911 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.563570 6400 scope.go:122] "RemoveContainer" containerID="06caa9d4b59c5fb5e93725c76a25920a462e9bbcf360460bdd58db0f8d7d60c7" Oct 30 17:15:19.564042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.563962 6400 scope.go:122] "RemoveContainer" containerID="e41235119f3e022efa87a80b3fb49d1367d5c151df3e3f1040db33492380fd44" Oct 30 17:15:19.564313 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:19.564285 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57_odh-ai-gateway-infra(91c62170-6668-4436-b794-10170604b7d0)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" podUID="91c62170-6668-4436-b794-10170604b7d0" Oct 30 17:15:19.565076 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.565050 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" Oct 30 17:15:19.565169 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.565083 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92" event={"ID":"75cea706-1357-4bf0-a987-6323bbf35317","Type":"ContainerDied","Data":"31fd118c74a0d50a53eef8976df1a129a4b04674a04d45160a682e1ad4a5f69f"} Oct 30 17:15:19.567009 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.566985 6400 generic.go:356] "Generic (PLEG): container finished" podID="510145d1-4707-4356-80de-09cf7da6540f" containerID="bded2cf2d0b2a7f178438bffd6b95ede9de51ccef6a76af11d753443cf69d2df" exitCode=60 Oct 30 17:15:19.567123 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.567061 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" event={"ID":"510145d1-4707-4356-80de-09cf7da6540f","Type":"ContainerDied","Data":"bded2cf2d0b2a7f178438bffd6b95ede9de51ccef6a76af11d753443cf69d2df"} Oct 30 17:15:19.567455 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.567436 6400 scope.go:122] "RemoveContainer" containerID="bded2cf2d0b2a7f178438bffd6b95ede9de51ccef6a76af11d753443cf69d2df" Oct 30 17:15:19.567794 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:19.567769 6400 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd_odh-ai-gateway-infra(510145d1-4707-4356-80de-09cf7da6540f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" podUID="510145d1-4707-4356-80de-09cf7da6540f" Oct 30 17:15:19.569880 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.569859 6400 generic.go:356] "Generic (PLEG): container finished" podID="e32f582c-5e90-4129-aa85-cdf31c99d0ac" containerID="b58492d331aaf3304dbdbc0774b144fef2cb11da4f56f1959180b51d56b04b1f" exitCode=0 Oct 30 17:15:19.569958 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.569922 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" event={"ID":"e32f582c-5e90-4129-aa85-cdf31c99d0ac","Type":"ContainerDied","Data":"b58492d331aaf3304dbdbc0774b144fef2cb11da4f56f1959180b51d56b04b1f"} Oct 30 17:15:19.577534 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.577516 6400 scope.go:122] "RemoveContainer" containerID="545cf66f68220880d6a610ed687560632016dc7a895b5e18e4449b1f3a6b9afc" Oct 30 17:15:19.587180 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.587147 6400 scope.go:122] "RemoveContainer" containerID="5e795a5a9779c3d36ac806232460109700fe2e5afab4facbdf7dc868c2160ea1" Oct 30 17:15:19.598427 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.598401 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57"] Oct 30 17:15:19.688777 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.688748 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd"] Oct 30 17:15:19.734770 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.734741 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92"] Oct 30 17:15:19.739344 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:19.739313 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-0b2348-29889675-w5x92"] Oct 30 17:15:20.787666 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.787632 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" Oct 30 17:15:20.822175 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.822153 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" Oct 30 17:15:20.832169 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.831913 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:20.878254 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.878214 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-xks-ca\") pod \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " Oct 30 17:15:20.878447 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.878284 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e32f582c-5e90-4129-aa85-cdf31c99d0ac-kube-api-access-nnqj2\" (UniqueName: \"kubernetes.io/projected/e32f582c-5e90-4129-aa85-cdf31c99d0ac-kube-api-access-nnqj2\") pod \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " Oct 30 17:15:20.878447 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.878425 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91c62170-6668-4436-b794-10170604b7d0-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c62170-6668-4436-b794-10170604b7d0-maas-api-service-ca\") pod \"91c62170-6668-4436-b794-10170604b7d0\" (UID: \"91c62170-6668-4436-b794-10170604b7d0\") " Oct 30 17:15:20.878566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.878455 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/510145d1-4707-4356-80de-09cf7da6540f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510145d1-4707-4356-80de-09cf7da6540f-maas-api-service-ca\") pod \"510145d1-4707-4356-80de-09cf7da6540f\" (UID: \"510145d1-4707-4356-80de-09cf7da6540f\") " Oct 30 17:15:20.878566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.878549 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/510145d1-4707-4356-80de-09cf7da6540f-kube-api-access-jvbwk\" (UniqueName: \"kubernetes.io/projected/510145d1-4707-4356-80de-09cf7da6540f-kube-api-access-jvbwk\") pod \"510145d1-4707-4356-80de-09cf7da6540f\" (UID: \"510145d1-4707-4356-80de-09cf7da6540f\") " Oct 30 17:15:20.878694 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.878649 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-openshift-ca\") pod \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\" (UID: \"e32f582c-5e90-4129-aa85-cdf31c99d0ac\") " Oct 30 17:15:20.878754 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.878705 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91c62170-6668-4436-b794-10170604b7d0-kube-api-access-qxvcb\" (UniqueName: \"kubernetes.io/projected/91c62170-6668-4436-b794-10170604b7d0-kube-api-access-qxvcb\") pod \"91c62170-6668-4436-b794-10170604b7d0\" (UID: \"91c62170-6668-4436-b794-10170604b7d0\") " Oct 30 17:15:20.879145 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.878832 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91c62170-6668-4436-b794-10170604b7d0-maas-api-service-ca" pod "91c62170-6668-4436-b794-10170604b7d0" (UID: "91c62170-6668-4436-b794-10170604b7d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:20.879145 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.879080 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-openshift-ca" pod "e32f582c-5e90-4129-aa85-cdf31c99d0ac" (UID: "e32f582c-5e90-4129-aa85-cdf31c99d0ac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:20.879346 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.879163 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/510145d1-4707-4356-80de-09cf7da6540f-maas-api-service-ca" pod "510145d1-4707-4356-80de-09cf7da6540f" (UID: "510145d1-4707-4356-80de-09cf7da6540f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:20.879346 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.879321 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91c62170-6668-4436-b794-10170604b7d0-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.879346 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.879342 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/510145d1-4707-4356-80de-09cf7da6540f-maas-api-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.879465 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.879357 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.880755 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.880725 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-xks-ca" pod "e32f582c-5e90-4129-aa85-cdf31c99d0ac" (UID: "e32f582c-5e90-4129-aa85-cdf31c99d0ac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:20.881054 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.881026 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e32f582c-5e90-4129-aa85-cdf31c99d0ac-kube-api-access-nnqj2" pod "e32f582c-5e90-4129-aa85-cdf31c99d0ac" (UID: "e32f582c-5e90-4129-aa85-cdf31c99d0ac"). InnerVolumeSpecName "kube-api-access-nnqj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:20.881244 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.881226 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91c62170-6668-4436-b794-10170604b7d0-kube-api-access-qxvcb" pod "91c62170-6668-4436-b794-10170604b7d0" (UID: "91c62170-6668-4436-b794-10170604b7d0"). InnerVolumeSpecName "kube-api-access-qxvcb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:20.881645 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.881581 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/510145d1-4707-4356-80de-09cf7da6540f-kube-api-access-jvbwk" pod "510145d1-4707-4356-80de-09cf7da6540f" (UID: "510145d1-4707-4356-80de-09cf7da6540f"). InnerVolumeSpecName "kube-api-access-jvbwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:20.968042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.968010 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75cea706-1357-4bf0-a987-6323bbf35317" path="/var/lib/kubelet/pods/75cea706-1357-4bf0-a987-6323bbf35317/volumes" Oct 30 17:15:20.981215 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.981191 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvbwk\" (UniqueName: \"kubernetes.io/projected/510145d1-4707-4356-80de-09cf7da6540f-kube-api-access-jvbwk\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.981321 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.981221 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxvcb\" (UniqueName: \"kubernetes.io/projected/91c62170-6668-4436-b794-10170604b7d0-kube-api-access-qxvcb\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.981321 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.981240 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e32f582c-5e90-4129-aa85-cdf31c99d0ac-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:20.981321 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:20.981253 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnqj2\" (UniqueName: \"kubernetes.io/projected/e32f582c-5e90-4129-aa85-cdf31c99d0ac-kube-api-access-nnqj2\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:21.591063 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.591029 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" Oct 30 17:15:21.591281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.591065 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57" event={"ID":"91c62170-6668-4436-b794-10170604b7d0","Type":"ContainerDied","Data":"59f5e8f98daa797f5949a74decfa9595e2b914471478526341dcd97513672e1f"} Oct 30 17:15:21.591281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.591139 6400 scope.go:122] "RemoveContainer" containerID="e41235119f3e022efa87a80b3fb49d1367d5c151df3e3f1040db33492380fd44" Oct 30 17:15:21.592786 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.592768 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" Oct 30 17:15:21.592860 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.592822 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd" event={"ID":"510145d1-4707-4356-80de-09cf7da6540f","Type":"ContainerDied","Data":"07ef145b4ca7e1edef8b9bcaa7366c7627502682af244d4948023dc1c0c3e029"} Oct 30 17:15:21.594946 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.594925 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" Oct 30 17:15:21.595067 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.594927 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z" event={"ID":"e32f582c-5e90-4129-aa85-cdf31c99d0ac","Type":"ContainerDied","Data":"6239d82ea3441b941d92c301adc818129697f5a8f0219b225d094d796ab48d66"} Oct 30 17:15:21.595067 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.594982 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6239d82ea3441b941d92c301adc818129697f5a8f0219b225d094d796ab48d66" Oct 30 17:15:21.603349 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.603324 6400 scope.go:122] "RemoveContainer" containerID="bded2cf2d0b2a7f178438bffd6b95ede9de51ccef6a76af11d753443cf69d2df" Oct 30 17:15:21.668936 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.668905 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57"] Oct 30 17:15:21.671927 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.671900 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-0ae3fe-29889675-fsl57"] Oct 30 17:15:21.707732 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.707697 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd"] Oct 30 17:15:21.713872 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:21.713849 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-ddc10d-29889675-pdxnd"] Oct 30 17:15:22.967386 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:22.967355 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="510145d1-4707-4356-80de-09cf7da6540f" path="/var/lib/kubelet/pods/510145d1-4707-4356-80de-09cf7da6540f/volumes" Oct 30 17:15:22.967852 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:22.967836 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91c62170-6668-4436-b794-10170604b7d0" path="/var/lib/kubelet/pods/91c62170-6668-4436-b794-10170604b7d0/volumes" Oct 30 17:15:24.600349 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:24.600307 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:15:28.949051 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:28.949021 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv"] Oct 30 17:15:28.949714 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:28.949228 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" podUID="bba32efe-e717-4821-996d-9da99af18277" containerName="maas-api" containerID="cri-o://e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888" gracePeriod=30 Oct 30 17:15:29.231462 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.231422 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 17:15:29.389319 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.389288 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bba32efe-e717-4821-996d-9da99af18277-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba32efe-e717-4821-996d-9da99af18277-openshift-service-ca\") pod \"bba32efe-e717-4821-996d-9da99af18277\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " Oct 30 17:15:29.389520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.389376 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-maas-api-tls\") pod \"bba32efe-e717-4821-996d-9da99af18277\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " Oct 30 17:15:29.389520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.389503 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bba32efe-e717-4821-996d-9da99af18277-kube-api-access-vmrf5\" (UniqueName: \"kubernetes.io/projected/bba32efe-e717-4821-996d-9da99af18277-kube-api-access-vmrf5\") pod \"bba32efe-e717-4821-996d-9da99af18277\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " Oct 30 17:15:29.389703 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.389675 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba32efe-e717-4821-996d-9da99af18277-openshift-service-ca" pod "bba32efe-e717-4821-996d-9da99af18277" (UID: "bba32efe-e717-4821-996d-9da99af18277"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:29.389775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.389687 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-metrics-certs\") pod \"bba32efe-e717-4821-996d-9da99af18277\" (UID: \"bba32efe-e717-4821-996d-9da99af18277\") " Oct 30 17:15:29.390418 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.390390 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba32efe-e717-4821-996d-9da99af18277-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:29.391500 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.391476 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-maas-api-tls" pod "bba32efe-e717-4821-996d-9da99af18277" (UID: "bba32efe-e717-4821-996d-9da99af18277"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:29.391743 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.391720 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-metrics-certs" pod "bba32efe-e717-4821-996d-9da99af18277" (UID: "bba32efe-e717-4821-996d-9da99af18277"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:29.391931 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.391911 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bba32efe-e717-4821-996d-9da99af18277-kube-api-access-vmrf5" pod "bba32efe-e717-4821-996d-9da99af18277" (UID: "bba32efe-e717-4821-996d-9da99af18277"). InnerVolumeSpecName "kube-api-access-vmrf5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:29.491803 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.491773 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:29.491803 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.491800 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bba32efe-e717-4821-996d-9da99af18277-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:29.491803 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.491810 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmrf5\" (UniqueName: \"kubernetes.io/projected/bba32efe-e717-4821-996d-9da99af18277-kube-api-access-vmrf5\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:29.656136 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.656101 6400 generic.go:356] "Generic (PLEG): container finished" podID="bba32efe-e717-4821-996d-9da99af18277" containerID="e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888" exitCode=0 Oct 30 17:15:29.656341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.656155 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" Oct 30 17:15:29.656341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.656189 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" event={"ID":"bba32efe-e717-4821-996d-9da99af18277","Type":"ContainerDied","Data":"e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888"} Oct 30 17:15:29.656341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.656275 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv" event={"ID":"bba32efe-e717-4821-996d-9da99af18277","Type":"ContainerDied","Data":"18991ab2473da218f2dde305502bd24589a82194ceddf6ab623e766a52a8076d"} Oct 30 17:15:29.656341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.656299 6400 scope.go:122] "RemoveContainer" containerID="e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888" Oct 30 17:15:29.668257 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.668236 6400 scope.go:122] "RemoveContainer" containerID="e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888" Oct 30 17:15:29.668535 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:29.668514 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888\": container with ID starting with e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888 not found: ID does not exist" containerID="e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888" Oct 30 17:15:29.668623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.668547 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888"} err="failed to get container status \"e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888\": rpc error: code = NotFound desc = could not find container \"e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888\": container with ID starting with e853c2ba1da672baa6c66d02029a3360ae0cf39deabbfb3735d29bf6c2b52888 not found: ID does not exist" Oct 30 17:15:29.715912 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.715872 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv"] Oct 30 17:15:29.720569 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:29.720541 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-3ed10a-5748f95d4f-h9nqv"] Oct 30 17:15:30.967174 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:30.967133 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bba32efe-e717-4821-996d-9da99af18277" path="/var/lib/kubelet/pods/bba32efe-e717-4821-996d-9da99af18277/volumes" Oct 30 17:15:31.325347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:31.325300 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm"] Oct 30 17:15:31.325667 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:31.325585 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" containerID="cri-o://c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946" gracePeriod=30 Oct 30 17:15:32.543666 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:32.543633 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:15:32.544003 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:32.543690 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:15:36.611186 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.611121 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 17:15:36.707008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.706974 6400 generic.go:356] "Generic (PLEG): container finished" podID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerID="c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946" exitCode=0 Oct 30 17:15:36.707229 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.707026 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" event={"ID":"a0a964e5-301d-4f97-b692-f2c9797276c4","Type":"ContainerDied","Data":"c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946"} Oct 30 17:15:36.707229 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.707042 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" Oct 30 17:15:36.707229 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.707071 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm" event={"ID":"a0a964e5-301d-4f97-b692-f2c9797276c4","Type":"ContainerDied","Data":"e98619113f239b9a761cc12425086395c4acba7cf896fa5c8b2ab06b836c6071"} Oct 30 17:15:36.707229 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.707095 6400 scope.go:122] "RemoveContainer" containerID="c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946" Oct 30 17:15:36.717535 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.717505 6400 scope.go:122] "RemoveContainer" containerID="300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb" Oct 30 17:15:36.727143 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.727126 6400 scope.go:122] "RemoveContainer" containerID="c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946" Oct 30 17:15:36.727409 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:36.727389 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946\": container with ID starting with c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946 not found: ID does not exist" containerID="c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946" Oct 30 17:15:36.727505 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.727420 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946"} err="failed to get container status \"c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946\": rpc error: code = NotFound desc = could not find container \"c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946\": container with ID starting with c9268efe49a62b3e53ec855d65d072cd017c5e11ae8690662253f4864ecd3946 not found: ID does not exist" Oct 30 17:15:36.727505 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.727436 6400 scope.go:122] "RemoveContainer" containerID="300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb" Oct 30 17:15:36.727711 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:36.727682 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb\": container with ID starting with 300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb not found: ID does not exist" containerID="300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb" Oct 30 17:15:36.727818 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.727718 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb"} err="failed to get container status \"300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb\": rpc error: code = NotFound desc = could not find container \"300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb\": container with ID starting with 300f66f90f64eb64db002e21675b70b8022e36ff2fe389e5ca6d3915f6f202cb not found: ID does not exist" Oct 30 17:15:36.776354 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776316 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-data\") pod \"a0a964e5-301d-4f97-b692-f2c9797276c4\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " Oct 30 17:15:36.776553 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776397 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-kube-api-access-hdldf\" (UniqueName: \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-kube-api-access-hdldf\") pod \"a0a964e5-301d-4f97-b692-f2c9797276c4\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " Oct 30 17:15:36.776553 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776422 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-envoy\") pod \"a0a964e5-301d-4f97-b692-f2c9797276c4\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " Oct 30 17:15:36.776553 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776490 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-token\" (UniqueName: \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-token\") pod \"a0a964e5-301d-4f97-b692-f2c9797276c4\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " Oct 30 17:15:36.776553 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776514 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-socket\") pod \"a0a964e5-301d-4f97-b692-f2c9797276c4\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " Oct 30 17:15:36.776873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776582 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-data" pod "a0a964e5-301d-4f97-b692-f2c9797276c4" (UID: "a0a964e5-301d-4f97-b692-f2c9797276c4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:36.776873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776680 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-certs\") pod \"a0a964e5-301d-4f97-b692-f2c9797276c4\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " Oct 30 17:15:36.776873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776735 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a964e5-301d-4f97-b692-f2c9797276c4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a964e5-301d-4f97-b692-f2c9797276c4-istiod-ca-cert\") pod \"a0a964e5-301d-4f97-b692-f2c9797276c4\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " Oct 30 17:15:36.776873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776797 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-podinfo\") pod \"a0a964e5-301d-4f97-b692-f2c9797276c4\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " Oct 30 17:15:36.776873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776843 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-socket" pod "a0a964e5-301d-4f97-b692-f2c9797276c4" (UID: "a0a964e5-301d-4f97-b692-f2c9797276c4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:36.777138 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.776889 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-credential-socket\") pod \"a0a964e5-301d-4f97-b692-f2c9797276c4\" (UID: \"a0a964e5-301d-4f97-b692-f2c9797276c4\") " Oct 30 17:15:36.777138 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.777128 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-certs" pod "a0a964e5-301d-4f97-b692-f2c9797276c4" (UID: "a0a964e5-301d-4f97-b692-f2c9797276c4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:36.777251 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.777178 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-credential-socket" pod "a0a964e5-301d-4f97-b692-f2c9797276c4" (UID: "a0a964e5-301d-4f97-b692-f2c9797276c4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:36.777251 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.777201 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a964e5-301d-4f97-b692-f2c9797276c4-istiod-ca-cert" pod "a0a964e5-301d-4f97-b692-f2c9797276c4" (UID: "a0a964e5-301d-4f97-b692-f2c9797276c4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:36.777549 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.777528 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:36.777643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.777553 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a0a964e5-301d-4f97-b692-f2c9797276c4-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:36.777643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.777566 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:36.777643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.777580 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:36.777643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.777593 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:36.779262 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.779230 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-podinfo" pod "a0a964e5-301d-4f97-b692-f2c9797276c4" (UID: "a0a964e5-301d-4f97-b692-f2c9797276c4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:36.779364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.779289 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-envoy" pod "a0a964e5-301d-4f97-b692-f2c9797276c4" (UID: "a0a964e5-301d-4f97-b692-f2c9797276c4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:36.779495 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.779476 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-kube-api-access-hdldf" pod "a0a964e5-301d-4f97-b692-f2c9797276c4" (UID: "a0a964e5-301d-4f97-b692-f2c9797276c4"). InnerVolumeSpecName "kube-api-access-hdldf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:36.779540 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.779495 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-token" pod "a0a964e5-301d-4f97-b692-f2c9797276c4" (UID: "a0a964e5-301d-4f97-b692-f2c9797276c4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:36.879297 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.879202 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdldf\" (UniqueName: \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-kube-api-access-hdldf\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:36.879297 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.879237 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:36.879297 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.879251 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:36.879297 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:36.879264 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a0a964e5-301d-4f97-b692-f2c9797276c4-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:37.051743 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:37.051711 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm"] Oct 30 17:15:37.064679 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:37.064646 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-dc74d5-openshift-default-76fc685fdc-nb8tm"] Oct 30 17:15:37.329748 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:37.329710 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q"] Oct 30 17:15:37.330289 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:37.330253 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerName="istio-proxy" containerID="cri-o://6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb" gracePeriod=30 Oct 30 17:15:38.453570 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:38.453532 6400 patch_prober.go:29] interesting pod/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 17:15:38.454065 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:38.453615 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 17:15:38.967073 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:38.967040 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" path="/var/lib/kubelet/pods/a0a964e5-301d-4f97-b692-f2c9797276c4/volumes" Oct 30 17:15:39.827725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:39.827682 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g"] Oct 30 17:15:39.828234 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:39.827951 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerName="istio-proxy" containerID="cri-o://26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f" gracePeriod=30 Oct 30 17:15:42.629852 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.629828 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:15:42.745835 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.745805 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-podinfo\") pod \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " Oct 30 17:15:42.746031 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.745862 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-data\") pod \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " Oct 30 17:15:42.746031 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.745887 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istiod-ca-cert\") pod \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " Oct 30 17:15:42.746031 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.745907 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-envoy\") pod \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " Oct 30 17:15:42.746031 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.745979 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-socket\") pod \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " Oct 30 17:15:42.746270 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746072 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-credential-socket\") pod \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " Oct 30 17:15:42.746270 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746126 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-certs\") pod \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " Oct 30 17:15:42.746270 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746191 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-data" pod "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" (UID: "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:42.746270 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746196 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-kube-api-access-brvgv\" (UniqueName: \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-kube-api-access-brvgv\") pod \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " Oct 30 17:15:42.746485 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746300 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-token\" (UniqueName: \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-token\") pod \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\" (UID: \"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1\") " Oct 30 17:15:42.746485 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746298 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istiod-ca-cert" pod "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" (UID: "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:42.746485 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746394 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-credential-socket" pod "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" (UID: "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:42.746485 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746410 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-socket" pod "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" (UID: "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:42.746742 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746583 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-certs" pod "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" (UID: "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:42.746853 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746835 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.746939 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746857 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.746939 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746871 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.746939 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746883 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.746939 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.746893 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.747594 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.747564 6400 generic.go:356] "Generic (PLEG): container finished" podID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerID="6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb" exitCode=0 Oct 30 17:15:42.747717 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.747688 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" Oct 30 17:15:42.747819 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.747796 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" event={"ID":"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1","Type":"ContainerDied","Data":"6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb"} Oct 30 17:15:42.747901 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.747886 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q" event={"ID":"c1e4f957-c66c-40dc-8560-a6bbd2cd75d1","Type":"ContainerDied","Data":"bdb5d6abfd2b69828e756a9ca0a803fee5f03848ee8b28b3667259bc53b998c4"} Oct 30 17:15:42.747956 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.747922 6400 scope.go:122] "RemoveContainer" containerID="6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb" Oct 30 17:15:42.749649 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.748865 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-kube-api-access-brvgv" pod "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" (UID: "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1"). InnerVolumeSpecName "kube-api-access-brvgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:42.749649 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.748982 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-token" pod "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" (UID: "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:42.749649 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.748981 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-podinfo" pod "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" (UID: "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:42.749649 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.749218 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-envoy" pod "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" (UID: "c1e4f957-c66c-40dc-8560-a6bbd2cd75d1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:42.775113 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.775090 6400 scope.go:122] "RemoveContainer" containerID="6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb" Oct 30 17:15:42.775407 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:42.775384 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb\": container with ID starting with 6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb not found: ID does not exist" containerID="6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb" Oct 30 17:15:42.775480 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.775415 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb"} err="failed to get container status \"6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb\": rpc error: code = NotFound desc = could not find container \"6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb\": container with ID starting with 6bcad7d92a7a3daa596f8755f5495fbb9b5a27846d463fb874252772a92b31eb not found: ID does not exist" Oct 30 17:15:42.848610 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.848569 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brvgv\" (UniqueName: \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-kube-api-access-brvgv\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.848847 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.848628 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.848847 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.848647 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:42.848847 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:42.848659 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:43.091007 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:43.090975 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q"] Oct 30 17:15:43.095095 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:43.095071 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-3ed10a-openshift-default-787f85d795-z847q"] Oct 30 17:15:44.967815 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:44.967779 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" path="/var/lib/kubelet/pods/c1e4f957-c66c-40dc-8560-a6bbd2cd75d1/volumes" Oct 30 17:15:45.086663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.086638 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:15:45.276038 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.275995 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-envoy\") pod \"05edb360-ea8e-4b72-846d-e3ecd317bb37\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " Oct 30 17:15:45.276256 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276055 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-token\" (UniqueName: \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-token\") pod \"05edb360-ea8e-4b72-846d-e3ecd317bb37\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " Oct 30 17:15:45.276256 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276120 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-data\") pod \"05edb360-ea8e-4b72-846d-e3ecd317bb37\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " Oct 30 17:15:45.276256 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276154 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-kube-api-access-g8b49\" (UniqueName: \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-kube-api-access-g8b49\") pod \"05edb360-ea8e-4b72-846d-e3ecd317bb37\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " Oct 30 17:15:45.276256 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276207 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-socket\") pod \"05edb360-ea8e-4b72-846d-e3ecd317bb37\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " Oct 30 17:15:45.276491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276272 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-credential-socket\") pod \"05edb360-ea8e-4b72-846d-e3ecd317bb37\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " Oct 30 17:15:45.276491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276299 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-certs\") pod \"05edb360-ea8e-4b72-846d-e3ecd317bb37\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " Oct 30 17:15:45.276491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276374 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05edb360-ea8e-4b72-846d-e3ecd317bb37-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05edb360-ea8e-4b72-846d-e3ecd317bb37-istiod-ca-cert\") pod \"05edb360-ea8e-4b72-846d-e3ecd317bb37\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " Oct 30 17:15:45.276491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276487 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-podinfo\") pod \"05edb360-ea8e-4b72-846d-e3ecd317bb37\" (UID: \"05edb360-ea8e-4b72-846d-e3ecd317bb37\") " Oct 30 17:15:45.276737 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276486 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-data" pod "05edb360-ea8e-4b72-846d-e3ecd317bb37" (UID: "05edb360-ea8e-4b72-846d-e3ecd317bb37"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:45.276737 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276660 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-socket" pod "05edb360-ea8e-4b72-846d-e3ecd317bb37" (UID: "05edb360-ea8e-4b72-846d-e3ecd317bb37"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:45.276844 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276807 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-certs" pod "05edb360-ea8e-4b72-846d-e3ecd317bb37" (UID: "05edb360-ea8e-4b72-846d-e3ecd317bb37"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:45.276945 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.276920 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05edb360-ea8e-4b72-846d-e3ecd317bb37-istiod-ca-cert" pod "05edb360-ea8e-4b72-846d-e3ecd317bb37" (UID: "05edb360-ea8e-4b72-846d-e3ecd317bb37"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:45.277387 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.277345 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-credential-socket" pod "05edb360-ea8e-4b72-846d-e3ecd317bb37" (UID: "05edb360-ea8e-4b72-846d-e3ecd317bb37"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:45.277513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.277387 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05edb360-ea8e-4b72-846d-e3ecd317bb37-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.277513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.277411 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.277513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.277425 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.277513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.277439 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.278746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.278723 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-kube-api-access-g8b49" pod "05edb360-ea8e-4b72-846d-e3ecd317bb37" (UID: "05edb360-ea8e-4b72-846d-e3ecd317bb37"). InnerVolumeSpecName "kube-api-access-g8b49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:45.278855 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.278760 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-token" pod "05edb360-ea8e-4b72-846d-e3ecd317bb37" (UID: "05edb360-ea8e-4b72-846d-e3ecd317bb37"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:45.278855 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.278793 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-envoy" pod "05edb360-ea8e-4b72-846d-e3ecd317bb37" (UID: "05edb360-ea8e-4b72-846d-e3ecd317bb37"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:15:45.279062 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.279046 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-podinfo" pod "05edb360-ea8e-4b72-846d-e3ecd317bb37" (UID: "05edb360-ea8e-4b72-846d-e3ecd317bb37"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:15:45.379131 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.379097 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.379131 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.379127 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.379131 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.379137 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8b49\" (UniqueName: \"kubernetes.io/projected/05edb360-ea8e-4b72-846d-e3ecd317bb37-kube-api-access-g8b49\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.379343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.379148 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05edb360-ea8e-4b72-846d-e3ecd317bb37-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.379343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.379158 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05edb360-ea8e-4b72-846d-e3ecd317bb37-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:45.772554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.772519 6400 generic.go:356] "Generic (PLEG): container finished" podID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerID="26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f" exitCode=0 Oct 30 17:15:45.772783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.772579 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" Oct 30 17:15:45.772783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.772584 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" event={"ID":"05edb360-ea8e-4b72-846d-e3ecd317bb37","Type":"ContainerDied","Data":"26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f"} Oct 30 17:15:45.772783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.772712 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g" event={"ID":"05edb360-ea8e-4b72-846d-e3ecd317bb37","Type":"ContainerDied","Data":"f31c463d319c02ae99ca152accd4a402fd07871fdbebd7e06f829890e94a0da6"} Oct 30 17:15:45.772783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.772733 6400 scope.go:122] "RemoveContainer" containerID="26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f" Oct 30 17:15:45.784457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.784438 6400 scope.go:122] "RemoveContainer" containerID="26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f" Oct 30 17:15:45.785076 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:45.785048 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f\": container with ID starting with 26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f not found: ID does not exist" containerID="26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f" Oct 30 17:15:45.785180 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.785077 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f"} err="failed to get container status \"26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f\": rpc error: code = NotFound desc = could not find container \"26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f\": container with ID starting with 26e7ccc81a185e6f6419805a3b71a895dbf2fe93245a746195bd3f89f685358f not found: ID does not exist" Oct 30 17:15:45.831286 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.831247 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g"] Oct 30 17:15:45.844070 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:45.844038 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-05810b-openshift-default-55f9d65747-dn97g"] Oct 30 17:15:46.967933 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:46.967898 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" path="/var/lib/kubelet/pods/05edb360-ea8e-4b72-846d-e3ecd317bb37/volumes" Oct 30 17:15:47.225910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.225827 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4"] Oct 30 17:15:47.226450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.226416 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" podUID="8978fa9f-b2c7-4132-a36f-69c9a4a9389c" containerName="maas-api" containerID="cri-o://10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652" gracePeriod=30 Oct 30 17:15:47.570713 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.570678 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:15:47.600400 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.600369 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-metrics-certs\") pod \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " Oct 30 17:15:47.600594 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.600428 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-maas-api-tls\") pod \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " Oct 30 17:15:47.600594 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.600485 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-kube-api-access-pp5s2\" (UniqueName: \"kubernetes.io/projected/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-kube-api-access-pp5s2\") pod \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " Oct 30 17:15:47.600594 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.600589 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-openshift-service-ca\") pod \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\" (UID: \"8978fa9f-b2c7-4132-a36f-69c9a4a9389c\") " Oct 30 17:15:47.600972 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.600946 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-openshift-service-ca" pod "8978fa9f-b2c7-4132-a36f-69c9a4a9389c" (UID: "8978fa9f-b2c7-4132-a36f-69c9a4a9389c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:15:47.601267 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.601198 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:47.602854 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.602829 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-maas-api-tls" pod "8978fa9f-b2c7-4132-a36f-69c9a4a9389c" (UID: "8978fa9f-b2c7-4132-a36f-69c9a4a9389c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:47.602938 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.602852 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-metrics-certs" pod "8978fa9f-b2c7-4132-a36f-69c9a4a9389c" (UID: "8978fa9f-b2c7-4132-a36f-69c9a4a9389c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:15:47.602938 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.602921 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-kube-api-access-pp5s2" pod "8978fa9f-b2c7-4132-a36f-69c9a4a9389c" (UID: "8978fa9f-b2c7-4132-a36f-69c9a4a9389c"). InnerVolumeSpecName "kube-api-access-pp5s2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:15:47.702636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.702556 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:47.702636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.702589 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:47.702636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.702640 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pp5s2\" (UniqueName: \"kubernetes.io/projected/8978fa9f-b2c7-4132-a36f-69c9a4a9389c-kube-api-access-pp5s2\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:15:47.789105 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.789065 6400 generic.go:356] "Generic (PLEG): container finished" podID="8978fa9f-b2c7-4132-a36f-69c9a4a9389c" containerID="10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652" exitCode=0 Oct 30 17:15:47.789281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.789115 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" Oct 30 17:15:47.789281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.789147 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" event={"ID":"8978fa9f-b2c7-4132-a36f-69c9a4a9389c","Type":"ContainerDied","Data":"10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652"} Oct 30 17:15:47.789281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.789204 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4" event={"ID":"8978fa9f-b2c7-4132-a36f-69c9a4a9389c","Type":"ContainerDied","Data":"0f95dd0a32fee0132936c1873eacc3bd3e16192928dded1918070b02d28dae47"} Oct 30 17:15:47.789281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.789230 6400 scope.go:122] "RemoveContainer" containerID="10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652" Oct 30 17:15:47.801074 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.801056 6400 scope.go:122] "RemoveContainer" containerID="10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652" Oct 30 17:15:47.801338 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:47.801320 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652\": container with ID starting with 10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652 not found: ID does not exist" containerID="10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652" Oct 30 17:15:47.801383 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.801347 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652"} err="failed to get container status \"10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652\": rpc error: code = NotFound desc = could not find container \"10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652\": container with ID starting with 10f11f0c88da4192e411be76432706f747e3dd0325c707cfe9082b21b5eb4652 not found: ID does not exist" Oct 30 17:15:47.843541 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.843510 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4"] Oct 30 17:15:47.848284 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.848244 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-0ae3fe-7947cf4664-qfsk4"] Oct 30 17:15:47.986710 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.986666 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j"] Oct 30 17:15:47.989175 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:47.989154 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-0ae3fe-qkb6j"] Oct 30 17:15:48.968163 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:48.968069 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d001992-9272-4b14-a48d-8468520e5975" path="/var/lib/kubelet/pods/4d001992-9272-4b14-a48d-8468520e5975/volumes" Oct 30 17:15:48.968676 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:48.968657 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8978fa9f-b2c7-4132-a36f-69c9a4a9389c" path="/var/lib/kubelet/pods/8978fa9f-b2c7-4132-a36f-69c9a4a9389c/volumes" Oct 30 17:15:54.290688 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:54.290652 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z"] Oct 30 17:15:54.303853 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:54.303819 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-97a47d-wbq2z"] Oct 30 17:15:54.599998 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:15:54.599903 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:15:54.966901 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:15:54.966819 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e32f582c-5e90-4129-aa85-cdf31c99d0ac" path="/var/lib/kubelet/pods/e32f582c-5e90-4129-aa85-cdf31c99d0ac/volumes" Oct 30 17:16:03.415458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.415422 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x"] Oct 30 17:16:03.416127 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.415673 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" podUID="80ef9893-a9f8-47b8-82cc-872188fbc467" containerName="maas-api" containerID="cri-o://21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078" gracePeriod=30 Oct 30 17:16:03.719847 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.717811 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 17:16:03.797934 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.797897 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-metrics-certs\") pod \"80ef9893-a9f8-47b8-82cc-872188fbc467\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " Oct 30 17:16:03.798138 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.797960 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80ef9893-a9f8-47b8-82cc-872188fbc467-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ef9893-a9f8-47b8-82cc-872188fbc467-openshift-service-ca\") pod \"80ef9893-a9f8-47b8-82cc-872188fbc467\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " Oct 30 17:16:03.798138 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.798055 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80ef9893-a9f8-47b8-82cc-872188fbc467-kube-api-access-tvzrd\" (UniqueName: \"kubernetes.io/projected/80ef9893-a9f8-47b8-82cc-872188fbc467-kube-api-access-tvzrd\") pod \"80ef9893-a9f8-47b8-82cc-872188fbc467\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " Oct 30 17:16:03.798138 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.798099 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-maas-api-tls\") pod \"80ef9893-a9f8-47b8-82cc-872188fbc467\" (UID: \"80ef9893-a9f8-47b8-82cc-872188fbc467\") " Oct 30 17:16:03.798386 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.798359 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80ef9893-a9f8-47b8-82cc-872188fbc467-openshift-service-ca" pod "80ef9893-a9f8-47b8-82cc-872188fbc467" (UID: "80ef9893-a9f8-47b8-82cc-872188fbc467"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:03.798665 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.798639 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/80ef9893-a9f8-47b8-82cc-872188fbc467-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:03.800484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.800453 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-maas-api-tls" pod "80ef9893-a9f8-47b8-82cc-872188fbc467" (UID: "80ef9893-a9f8-47b8-82cc-872188fbc467"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:03.800484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.800465 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-metrics-certs" pod "80ef9893-a9f8-47b8-82cc-872188fbc467" (UID: "80ef9893-a9f8-47b8-82cc-872188fbc467"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:03.800684 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.800521 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80ef9893-a9f8-47b8-82cc-872188fbc467-kube-api-access-tvzrd" pod "80ef9893-a9f8-47b8-82cc-872188fbc467" (UID: "80ef9893-a9f8-47b8-82cc-872188fbc467"). InnerVolumeSpecName "kube-api-access-tvzrd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:03.899822 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.899785 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvzrd\" (UniqueName: \"kubernetes.io/projected/80ef9893-a9f8-47b8-82cc-872188fbc467-kube-api-access-tvzrd\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:03.899822 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.899817 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:03.899822 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.899827 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/80ef9893-a9f8-47b8-82cc-872188fbc467-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:03.910455 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.910416 6400 generic.go:356] "Generic (PLEG): container finished" podID="80ef9893-a9f8-47b8-82cc-872188fbc467" containerID="21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078" exitCode=0 Oct 30 17:16:03.910728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.910474 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" Oct 30 17:16:03.910728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.910503 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" event={"ID":"80ef9893-a9f8-47b8-82cc-872188fbc467","Type":"ContainerDied","Data":"21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078"} Oct 30 17:16:03.910728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.910578 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x" event={"ID":"80ef9893-a9f8-47b8-82cc-872188fbc467","Type":"ContainerDied","Data":"c42346a0626c31d6842ce17cc87c13d90e019ea1c5dc0337d57c7e989c4950d4"} Oct 30 17:16:03.910728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.910633 6400 scope.go:122] "RemoveContainer" containerID="21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078" Oct 30 17:16:03.928147 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.928125 6400 scope.go:122] "RemoveContainer" containerID="21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078" Oct 30 17:16:03.928484 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:16:03.928458 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078\": container with ID starting with 21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078 not found: ID does not exist" containerID="21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078" Oct 30 17:16:03.928554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.928501 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078"} err="failed to get container status \"21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078\": rpc error: code = NotFound desc = could not find container \"21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078\": container with ID starting with 21c441ff56ee967ab878537f9bb167674995828faf18563dd29c1a74cfe91078 not found: ID does not exist" Oct 30 17:16:03.966425 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.966351 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x"] Oct 30 17:16:03.971276 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:03.971249 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-661509-6b6d4f865d-88x6x"] Oct 30 17:16:04.966667 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:04.966637 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80ef9893-a9f8-47b8-82cc-872188fbc467" path="/var/lib/kubelet/pods/80ef9893-a9f8-47b8-82cc-872188fbc467/volumes" Oct 30 17:16:05.782226 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:05.782189 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh"] Oct 30 17:16:05.782417 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:05.782395 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" podUID="1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" containerName="maas-api" containerID="cri-o://cc14a57ffb5ca27bded6a6882872eea299e5b1bfca18cf57681e666bbdde2ffa" gracePeriod=30 Oct 30 17:16:05.928656 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:05.928622 6400 generic.go:356] "Generic (PLEG): container finished" podID="1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" containerID="cc14a57ffb5ca27bded6a6882872eea299e5b1bfca18cf57681e666bbdde2ffa" exitCode=0 Oct 30 17:16:05.928656 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:05.928637 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" event={"ID":"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d","Type":"ContainerDied","Data":"cc14a57ffb5ca27bded6a6882872eea299e5b1bfca18cf57681e666bbdde2ffa"} Oct 30 17:16:05.942002 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:05.941969 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" podUID="1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.177:8443/health\": dial tcp 10.134.0.177:8443: connect: connection refused" Oct 30 17:16:06.091336 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.091307 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:16:06.227581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.227548 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-kube-api-access-vsrm7\" (UniqueName: \"kubernetes.io/projected/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-kube-api-access-vsrm7\") pod \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " Oct 30 17:16:06.227764 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.227692 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-maas-api-tls\") pod \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " Oct 30 17:16:06.227764 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.227731 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-openshift-service-ca\") pod \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " Oct 30 17:16:06.227846 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.227821 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-metrics-certs\") pod \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\" (UID: \"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d\") " Oct 30 17:16:06.228212 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.228186 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-openshift-service-ca" pod "1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" (UID: "1e2923a4-b7a1-49b9-95b1-1ff146f5f39d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:06.228354 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.228306 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:06.229897 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.229875 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-kube-api-access-vsrm7" pod "1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" (UID: "1e2923a4-b7a1-49b9-95b1-1ff146f5f39d"). InnerVolumeSpecName "kube-api-access-vsrm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:06.229994 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.229978 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-maas-api-tls" pod "1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" (UID: "1e2923a4-b7a1-49b9-95b1-1ff146f5f39d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:06.230063 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.230050 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-metrics-certs" pod "1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" (UID: "1e2923a4-b7a1-49b9-95b1-1ff146f5f39d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:06.329846 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.329764 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsrm7\" (UniqueName: \"kubernetes.io/projected/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-kube-api-access-vsrm7\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:06.329846 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.329789 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:06.329846 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.329799 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:06.332548 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.332517 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq"] Oct 30 17:16:06.385170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.385135 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j"] Oct 30 17:16:06.936422 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.936394 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" Oct 30 17:16:06.936597 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.936422 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh" event={"ID":"1e2923a4-b7a1-49b9-95b1-1ff146f5f39d","Type":"ContainerDied","Data":"02cd90835b05b0cf8e8f2f61a55d26cb102212b8e9dde64464f9e35c214407ab"} Oct 30 17:16:06.936597 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.936479 6400 scope.go:122] "RemoveContainer" containerID="cc14a57ffb5ca27bded6a6882872eea299e5b1bfca18cf57681e666bbdde2ffa" Oct 30 17:16:06.996281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:06.996253 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh"] Oct 30 17:16:07.009558 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:07.009531 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-de18d5-6cc56988d8-47vrh"] Oct 30 17:16:08.966864 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:08.966830 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" path="/var/lib/kubelet/pods/1e2923a4-b7a1-49b9-95b1-1ff146f5f39d/volumes" Oct 30 17:16:11.333300 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:11.333240 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" podUID="52ba393f-b73e-4e8e-b509-12dabb21e851" containerName="payload-processing" containerID="cri-o://dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353" gracePeriod=25 Oct 30 17:16:11.385691 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:11.385583 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" podUID="f28e6474-6a1e-4727-b3c4-9ea757a7ea3d" containerName="payload-pre-processing" containerID="cri-o://d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3" gracePeriod=25 Oct 30 17:16:11.886302 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:11.886267 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd"] Oct 30 17:16:11.886557 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:11.886531 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" podUID="a7e415d4-b0b2-40c1-b9fe-beb23f9c0809" containerName="maas-api" containerID="cri-o://753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0" gracePeriod=30 Oct 30 17:16:12.168820 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.168792 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 17:16:12.196174 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.196140 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-openshift-service-ca\") pod \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " Oct 30 17:16:12.196377 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.196237 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-maas-api-tls\") pod \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " Oct 30 17:16:12.196377 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.196343 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-kube-api-access-9449l\" (UniqueName: \"kubernetes.io/projected/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-kube-api-access-9449l\") pod \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " Oct 30 17:16:12.196503 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.196406 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-metrics-certs\") pod \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\" (UID: \"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809\") " Oct 30 17:16:12.196621 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.196573 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-openshift-service-ca" pod "a7e415d4-b0b2-40c1-b9fe-beb23f9c0809" (UID: "a7e415d4-b0b2-40c1-b9fe-beb23f9c0809"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:12.197020 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.196925 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:12.198792 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.198762 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-metrics-certs" pod "a7e415d4-b0b2-40c1-b9fe-beb23f9c0809" (UID: "a7e415d4-b0b2-40c1-b9fe-beb23f9c0809"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:12.198792 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.198777 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-maas-api-tls" pod "a7e415d4-b0b2-40c1-b9fe-beb23f9c0809" (UID: "a7e415d4-b0b2-40c1-b9fe-beb23f9c0809"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:12.198792 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.198760 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-kube-api-access-9449l" pod "a7e415d4-b0b2-40c1-b9fe-beb23f9c0809" (UID: "a7e415d4-b0b2-40c1-b9fe-beb23f9c0809"). InnerVolumeSpecName "kube-api-access-9449l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:12.298747 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.298699 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:12.298747 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.298743 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:12.298747 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.298753 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9449l\" (UniqueName: \"kubernetes.io/projected/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809-kube-api-access-9449l\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:12.978448 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.978414 6400 generic.go:356] "Generic (PLEG): container finished" podID="a7e415d4-b0b2-40c1-b9fe-beb23f9c0809" containerID="753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0" exitCode=0 Oct 30 17:16:12.978845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.978456 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" Oct 30 17:16:12.978845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.978490 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" event={"ID":"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809","Type":"ContainerDied","Data":"753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0"} Oct 30 17:16:12.978845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.978541 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd" event={"ID":"a7e415d4-b0b2-40c1-b9fe-beb23f9c0809","Type":"ContainerDied","Data":"1ce9e1af2fb8a848ec2078b60482ee357040997c266a2a1d0e01f6c607a1e94b"} Oct 30 17:16:12.978845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.978565 6400 scope.go:122] "RemoveContainer" containerID="753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0" Oct 30 17:16:12.989508 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.989492 6400 scope.go:122] "RemoveContainer" containerID="753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0" Oct 30 17:16:12.989804 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:16:12.989783 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0\": container with ID starting with 753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0 not found: ID does not exist" containerID="753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0" Oct 30 17:16:12.989888 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:12.989813 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0"} err="failed to get container status \"753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0\": rpc error: code = NotFound desc = could not find container \"753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0\": container with ID starting with 753f2b9b7cdf3148cca8dbf278042a697aacb98fa87b7eef678d45c51631b2b0 not found: ID does not exist" Oct 30 17:16:13.034280 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:13.034241 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd"] Oct 30 17:16:13.047404 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:13.047378 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-7c4b84-7fb86d4b4-lmzjd"] Oct 30 17:16:14.967082 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:14.967051 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a7e415d4-b0b2-40c1-b9fe-beb23f9c0809" path="/var/lib/kubelet/pods/a7e415d4-b0b2-40c1-b9fe-beb23f9c0809/volumes" Oct 30 17:16:17.152486 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:17.152449 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z"] Oct 30 17:16:17.792365 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:17.792328 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" podUID="52ba393f-b73e-4e8e-b509-12dabb21e851" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:18.799641 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:18.799557 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" podUID="f28e6474-6a1e-4727-b3c4-9ea757a7ea3d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:24.600769 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:16:24.600731 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:16:26.403221 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:26.403183 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz"] Oct 30 17:16:26.416281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:26.416252 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v"] Oct 30 17:16:27.792423 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:27.792388 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" podUID="52ba393f-b73e-4e8e-b509-12dabb21e851" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:28.798859 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:28.798819 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" podUID="f28e6474-6a1e-4727-b3c4-9ea757a7ea3d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 17:16:31.671580 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.671552 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:16:31.698691 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.698666 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:16:31.835346 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.835309 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-plugins-config-volume\") pod \"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d\" (UID: \"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d\") " Oct 30 17:16:31.835346 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.835353 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ba393f-b73e-4e8e-b509-12dabb21e851-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ba393f-b73e-4e8e-b509-12dabb21e851-plugins-config-volume\") pod \"52ba393f-b73e-4e8e-b509-12dabb21e851\" (UID: \"52ba393f-b73e-4e8e-b509-12dabb21e851\") " Oct 30 17:16:31.835645 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.835381 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ba393f-b73e-4e8e-b509-12dabb21e851-kube-api-access-f72c6\" (UniqueName: \"kubernetes.io/projected/52ba393f-b73e-4e8e-b509-12dabb21e851-kube-api-access-f72c6\") pod \"52ba393f-b73e-4e8e-b509-12dabb21e851\" (UID: \"52ba393f-b73e-4e8e-b509-12dabb21e851\") " Oct 30 17:16:31.835645 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.835433 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-kube-api-access-75995\" (UniqueName: \"kubernetes.io/projected/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-kube-api-access-75995\") pod \"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d\" (UID: \"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d\") " Oct 30 17:16:31.835798 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.835769 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-plugins-config-volume" pod "f28e6474-6a1e-4727-b3c4-9ea757a7ea3d" (UID: "f28e6474-6a1e-4727-b3c4-9ea757a7ea3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:31.835853 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.835811 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ba393f-b73e-4e8e-b509-12dabb21e851-plugins-config-volume" pod "52ba393f-b73e-4e8e-b509-12dabb21e851" (UID: "52ba393f-b73e-4e8e-b509-12dabb21e851"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:31.836039 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.836022 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-plugins-config-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:31.836099 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.836043 6400 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ba393f-b73e-4e8e-b509-12dabb21e851-plugins-config-volume\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:31.837790 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.837768 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ba393f-b73e-4e8e-b509-12dabb21e851-kube-api-access-f72c6" pod "52ba393f-b73e-4e8e-b509-12dabb21e851" (UID: "52ba393f-b73e-4e8e-b509-12dabb21e851"). InnerVolumeSpecName "kube-api-access-f72c6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:31.837931 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.837910 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-kube-api-access-75995" pod "f28e6474-6a1e-4727-b3c4-9ea757a7ea3d" (UID: "f28e6474-6a1e-4727-b3c4-9ea757a7ea3d"). InnerVolumeSpecName "kube-api-access-75995". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:31.936819 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.936735 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f72c6\" (UniqueName: \"kubernetes.io/projected/52ba393f-b73e-4e8e-b509-12dabb21e851-kube-api-access-f72c6\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:31.936819 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:31.936761 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75995\" (UniqueName: \"kubernetes.io/projected/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d-kube-api-access-75995\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.099463 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.099429 6400 generic.go:356] "Generic (PLEG): container finished" podID="52ba393f-b73e-4e8e-b509-12dabb21e851" containerID="dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353" exitCode=0 Oct 30 17:16:32.099675 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.099493 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" Oct 30 17:16:32.099675 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.099506 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" event={"ID":"52ba393f-b73e-4e8e-b509-12dabb21e851","Type":"ContainerDied","Data":"dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353"} Oct 30 17:16:32.099675 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.099595 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq" event={"ID":"52ba393f-b73e-4e8e-b509-12dabb21e851","Type":"ContainerDied","Data":"2c648b950dc0f0a7594fabaafd6f147e402ad5b188fff0b91138f0a227e7a0ce"} Oct 30 17:16:32.100060 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.100042 6400 scope.go:122] "RemoveContainer" containerID="dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353" Oct 30 17:16:32.102166 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.102056 6400 generic.go:356] "Generic (PLEG): container finished" podID="f28e6474-6a1e-4727-b3c4-9ea757a7ea3d" containerID="d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3" exitCode=0 Oct 30 17:16:32.102166 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.102113 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" event={"ID":"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d","Type":"ContainerDied","Data":"d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3"} Oct 30 17:16:32.102166 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.102148 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" event={"ID":"f28e6474-6a1e-4727-b3c4-9ea757a7ea3d","Type":"ContainerDied","Data":"9570c435ad3a0d6907589b18c27547a74a885f4ef7b32fd783d84bc713c334bc"} Oct 30 17:16:32.102357 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.102211 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j" Oct 30 17:16:32.115492 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.115468 6400 scope.go:122] "RemoveContainer" containerID="dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353" Oct 30 17:16:32.115821 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:16:32.115801 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353\": container with ID starting with dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353 not found: ID does not exist" containerID="dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353" Oct 30 17:16:32.115928 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.115828 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353"} err="failed to get container status \"dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353\": rpc error: code = NotFound desc = could not find container \"dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353\": container with ID starting with dbb387b726cda56dc8a2d5c37ad28edeb43db8404054771774c92497675cf353 not found: ID does not exist" Oct 30 17:16:32.115928 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.115843 6400 scope.go:122] "RemoveContainer" containerID="d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3" Oct 30 17:16:32.126454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.126428 6400 scope.go:122] "RemoveContainer" containerID="d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3" Oct 30 17:16:32.126777 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:16:32.126756 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3\": container with ID starting with d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3 not found: ID does not exist" containerID="d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3" Oct 30 17:16:32.126884 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.126784 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3"} err="failed to get container status \"d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3\": rpc error: code = NotFound desc = could not find container \"d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3\": container with ID starting with d0c1fee55d7dc84000f2d4514e7258122256fcbf36ae984c3868c1a894bf38c3 not found: ID does not exist" Oct 30 17:16:32.155908 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.155874 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq"] Oct 30 17:16:32.169088 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.169057 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-b4214a-7d9476b9c8-c82xq"] Oct 30 17:16:32.173918 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.173892 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" containerID="cri-o://046785d25e24219b5687975012a2edd9135cc8cf93d378583f266acf4c1079b7" gracePeriod=45 Oct 30 17:16:32.196651 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.196582 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j"] Oct 30 17:16:32.200170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.200148 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-b4214a-79794fdc87-tlm6j"] Oct 30 17:16:32.458375 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.458313 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 17:16:32.521049 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.521013 6400 scope.go:122] "RemoveContainer" containerID="046785d25e24219b5687975012a2edd9135cc8cf93d378583f266acf4c1079b7" Oct 30 17:16:32.644438 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.644404 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tls-certs\") pod \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " Oct 30 17:16:32.644666 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.644493 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-dshm\") pod \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " Oct 30 17:16:32.644666 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.644563 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-model-cache\") pod \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " Oct 30 17:16:32.644666 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.644656 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-kube-api-access-k46sh\" (UniqueName: \"kubernetes.io/projected/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-kube-api-access-k46sh\") pod \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " Oct 30 17:16:32.644854 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.644700 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tmp-dir\") pod \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " Oct 30 17:16:32.644854 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.644771 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-home\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-home\") pod \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\" (UID: \"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7\") " Oct 30 17:16:32.644983 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.644863 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-model-cache" pod "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" (UID: "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.645098 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.645075 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tmp-dir" pod "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" (UID: "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.645361 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.645341 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-home" pod "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" (UID: "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.645459 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.645354 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.645459 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.645377 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.647063 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.647034 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tls-certs" pod "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" (UID: "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:32.647233 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.647214 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-kube-api-access-k46sh" pod "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" (UID: "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7"). InnerVolumeSpecName "kube-api-access-k46sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:32.647417 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.647401 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-dshm" pod "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" (UID: "6c1fd2b9-f051-4cbe-a305-57d6136e2cf7"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:32.746233 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.746203 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.746233 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.746235 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k46sh\" (UniqueName: \"kubernetes.io/projected/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-kube-api-access-k46sh\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.746669 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.746250 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.746669 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.746264 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:32.968277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.968239 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ba393f-b73e-4e8e-b509-12dabb21e851" path="/var/lib/kubelet/pods/52ba393f-b73e-4e8e-b509-12dabb21e851/volumes" Oct 30 17:16:32.968863 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:32.968818 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28e6474-6a1e-4727-b3c4-9ea757a7ea3d" path="/var/lib/kubelet/pods/f28e6474-6a1e-4727-b3c4-9ea757a7ea3d/volumes" Oct 30 17:16:33.108460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:33.108382 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" event={"ID":"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7","Type":"ContainerDied","Data":"046785d25e24219b5687975012a2edd9135cc8cf93d378583f266acf4c1079b7"} Oct 30 17:16:33.108460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:33.108429 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" event={"ID":"6c1fd2b9-f051-4cbe-a305-57d6136e2cf7","Type":"ContainerDied","Data":"7c4d7bb735df152c51ef91a4fea7995ff5f6a5f8b5e5287abf36233f09e0b19e"} Oct 30 17:16:33.108698 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:33.108482 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" Oct 30 17:16:33.189408 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:33.189368 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z"] Oct 30 17:16:33.193137 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:33.193107 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z"] Oct 30 17:16:33.444739 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:33.444642 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-0b2348/simulator-w3-0b2348-kserve-7b5cb55c4f-6rl2z" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded" Oct 30 17:16:34.968421 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:34.968383 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" path="/var/lib/kubelet/pods/6c1fd2b9-f051-4cbe-a305-57d6136e2cf7/volumes" Oct 30 17:16:36.156471 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.156431 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd"] Oct 30 17:16:36.157002 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.156914 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" podUID="641a877c-4c86-4e4c-b331-c8f71e8760a1" containerName="maas-api" containerID="cri-o://ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466" gracePeriod=30 Oct 30 17:16:36.479240 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.478918 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:16:36.492723 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.492677 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-maas-api-tls\") pod \"641a877c-4c86-4e4c-b331-c8f71e8760a1\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " Oct 30 17:16:36.492913 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.492746 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/641a877c-4c86-4e4c-b331-c8f71e8760a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/641a877c-4c86-4e4c-b331-c8f71e8760a1-openshift-service-ca\") pod \"641a877c-4c86-4e4c-b331-c8f71e8760a1\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " Oct 30 17:16:36.492913 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.492822 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-metrics-certs\") pod \"641a877c-4c86-4e4c-b331-c8f71e8760a1\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " Oct 30 17:16:36.493055 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.492911 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/641a877c-4c86-4e4c-b331-c8f71e8760a1-kube-api-access-ffxcr\" (UniqueName: \"kubernetes.io/projected/641a877c-4c86-4e4c-b331-c8f71e8760a1-kube-api-access-ffxcr\") pod \"641a877c-4c86-4e4c-b331-c8f71e8760a1\" (UID: \"641a877c-4c86-4e4c-b331-c8f71e8760a1\") " Oct 30 17:16:36.493543 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.493144 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/641a877c-4c86-4e4c-b331-c8f71e8760a1-openshift-service-ca" pod "641a877c-4c86-4e4c-b331-c8f71e8760a1" (UID: "641a877c-4c86-4e4c-b331-c8f71e8760a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:36.493543 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.493457 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/641a877c-4c86-4e4c-b331-c8f71e8760a1-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:36.495727 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.495699 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-metrics-certs" pod "641a877c-4c86-4e4c-b331-c8f71e8760a1" (UID: "641a877c-4c86-4e4c-b331-c8f71e8760a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:36.497054 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.497026 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/641a877c-4c86-4e4c-b331-c8f71e8760a1-kube-api-access-ffxcr" pod "641a877c-4c86-4e4c-b331-c8f71e8760a1" (UID: "641a877c-4c86-4e4c-b331-c8f71e8760a1"). InnerVolumeSpecName "kube-api-access-ffxcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:36.497175 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.497057 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-maas-api-tls" pod "641a877c-4c86-4e4c-b331-c8f71e8760a1" (UID: "641a877c-4c86-4e4c-b331-c8f71e8760a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:36.594710 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.594681 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:36.594710 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.594709 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ffxcr\" (UniqueName: \"kubernetes.io/projected/641a877c-4c86-4e4c-b331-c8f71e8760a1-kube-api-access-ffxcr\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:36.594909 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:36.594719 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/641a877c-4c86-4e4c-b331-c8f71e8760a1-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:37.145245 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:37.145214 6400 generic.go:356] "Generic (PLEG): container finished" podID="641a877c-4c86-4e4c-b331-c8f71e8760a1" containerID="ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466" exitCode=0 Oct 30 17:16:37.145433 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:37.145256 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" event={"ID":"641a877c-4c86-4e4c-b331-c8f71e8760a1","Type":"ContainerDied","Data":"ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466"} Oct 30 17:16:37.145433 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:37.145265 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" Oct 30 17:16:37.145433 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:37.145297 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd" event={"ID":"641a877c-4c86-4e4c-b331-c8f71e8760a1","Type":"ContainerDied","Data":"d0e34a2f9dd31514076eb8edf56eba57e62cffd10902c432f4e8b925c9699ce7"} Oct 30 17:16:37.145433 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:37.145317 6400 scope.go:122] "RemoveContainer" containerID="ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466" Oct 30 17:16:37.155677 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:37.155649 6400 scope.go:122] "RemoveContainer" containerID="ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466" Oct 30 17:16:37.156114 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:16:37.156089 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466\": container with ID starting with ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466 not found: ID does not exist" containerID="ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466" Oct 30 17:16:37.156219 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:37.156126 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466"} err="failed to get container status \"ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466\": rpc error: code = NotFound desc = could not find container \"ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466\": container with ID starting with ef05cf00e0b54d5d4b2001218b10bdb8f61e0ef3466616bcdbaac1b1fb024466 not found: ID does not exist" Oct 30 17:16:37.203127 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:37.203088 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd"] Oct 30 17:16:37.216476 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:37.216442 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-ddc10d-58d7bb9f94-c8vkd"] Oct 30 17:16:38.968461 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:38.968427 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="641a877c-4c86-4e4c-b331-c8f71e8760a1" path="/var/lib/kubelet/pods/641a877c-4c86-4e4c-b331-c8f71e8760a1/volumes" Oct 30 17:16:41.425710 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.425671 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" containerID="cri-o://bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16" gracePeriod=45 Oct 30 17:16:41.437155 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.437112 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" containerID="cri-o://6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278" gracePeriod=45 Oct 30 17:16:41.572041 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.571995 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": dial tcp 10.134.0.50:8000: connect: connection refused" Oct 30 17:16:41.710038 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.710013 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 17:16:41.712825 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.712803 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 17:16:41.749908 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.749880 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19ea2eab-3e12-4193-8f63-aa7872b284c2-tls-certs\" (UniqueName: \"kubernetes.io/secret/19ea2eab-3e12-4193-8f63-aa7872b284c2-tls-certs\") pod \"19ea2eab-3e12-4193-8f63-aa7872b284c2\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " Oct 30 17:16:41.750096 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.749945 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0c5e6d3-f198-4daa-891f-6af1994e5d89-kube-api-access-dbc7l\" (UniqueName: \"kubernetes.io/projected/f0c5e6d3-f198-4daa-891f-6af1994e5d89-kube-api-access-dbc7l\") pod \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " Oct 30 17:16:41.750096 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.749974 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-home\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-home\") pod \"19ea2eab-3e12-4193-8f63-aa7872b284c2\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " Oct 30 17:16:41.750225 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750180 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-tmp-dir\") pod \"19ea2eab-3e12-4193-8f63-aa7872b284c2\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " Oct 30 17:16:41.750295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750241 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-dshm\") pod \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " Oct 30 17:16:41.750295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750261 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-home" pod "19ea2eab-3e12-4193-8f63-aa7872b284c2" (UID: "19ea2eab-3e12-4193-8f63-aa7872b284c2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.750295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750282 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-model-cache\") pod \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " Oct 30 17:16:41.750460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750314 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tls-certs\") pod \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " Oct 30 17:16:41.750460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750392 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tmp-dir\") pod \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " Oct 30 17:16:41.750460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750422 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-model-cache\") pod \"19ea2eab-3e12-4193-8f63-aa7872b284c2\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " Oct 30 17:16:41.750460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750429 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-tmp-dir" pod "19ea2eab-3e12-4193-8f63-aa7872b284c2" (UID: "19ea2eab-3e12-4193-8f63-aa7872b284c2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.750460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750456 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ea2eab-3e12-4193-8f63-aa7872b284c2-kube-api-access-vzwjn\" (UniqueName: \"kubernetes.io/projected/19ea2eab-3e12-4193-8f63-aa7872b284c2-kube-api-access-vzwjn\") pod \"19ea2eab-3e12-4193-8f63-aa7872b284c2\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " Oct 30 17:16:41.750785 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750501 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-home\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-home\") pod \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\" (UID: \"f0c5e6d3-f198-4daa-891f-6af1994e5d89\") " Oct 30 17:16:41.750785 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750528 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-dshm\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-dshm\") pod \"19ea2eab-3e12-4193-8f63-aa7872b284c2\" (UID: \"19ea2eab-3e12-4193-8f63-aa7872b284c2\") " Oct 30 17:16:41.750985 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750806 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tmp-dir" pod "f0c5e6d3-f198-4daa-891f-6af1994e5d89" (UID: "f0c5e6d3-f198-4daa-891f-6af1994e5d89"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.750985 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.750844 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-model-cache" pod "f0c5e6d3-f198-4daa-891f-6af1994e5d89" (UID: "f0c5e6d3-f198-4daa-891f-6af1994e5d89"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.751106 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.751061 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-home" pod "f0c5e6d3-f198-4daa-891f-6af1994e5d89" (UID: "f0c5e6d3-f198-4daa-891f-6af1994e5d89"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.752181 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.751258 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.752181 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.751309 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.752181 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.751325 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.752181 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.751337 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.752181 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.751348 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.752181 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.752145 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-model-cache" pod "19ea2eab-3e12-4193-8f63-aa7872b284c2" (UID: "19ea2eab-3e12-4193-8f63-aa7872b284c2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.758128 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.754638 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19ea2eab-3e12-4193-8f63-aa7872b284c2-tls-certs" pod "19ea2eab-3e12-4193-8f63-aa7872b284c2" (UID: "19ea2eab-3e12-4193-8f63-aa7872b284c2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:41.758637 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.758593 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-dshm" pod "f0c5e6d3-f198-4daa-891f-6af1994e5d89" (UID: "f0c5e6d3-f198-4daa-891f-6af1994e5d89"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.764006 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.760307 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-dshm" pod "19ea2eab-3e12-4193-8f63-aa7872b284c2" (UID: "19ea2eab-3e12-4193-8f63-aa7872b284c2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:16:41.764006 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.760555 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ea2eab-3e12-4193-8f63-aa7872b284c2-kube-api-access-vzwjn" pod "19ea2eab-3e12-4193-8f63-aa7872b284c2" (UID: "19ea2eab-3e12-4193-8f63-aa7872b284c2"). InnerVolumeSpecName "kube-api-access-vzwjn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:41.764006 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.761383 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tls-certs" pod "f0c5e6d3-f198-4daa-891f-6af1994e5d89" (UID: "f0c5e6d3-f198-4daa-891f-6af1994e5d89"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:41.768927 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.768902 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c5e6d3-f198-4daa-891f-6af1994e5d89-kube-api-access-dbc7l" pod "f0c5e6d3-f198-4daa-891f-6af1994e5d89" (UID: "f0c5e6d3-f198-4daa-891f-6af1994e5d89"). InnerVolumeSpecName "kube-api-access-dbc7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:41.852862 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.852830 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0c5e6d3-f198-4daa-891f-6af1994e5d89-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.852862 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.852858 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0c5e6d3-f198-4daa-891f-6af1994e5d89-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.853074 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.852869 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.853074 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.852888 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzwjn\" (UniqueName: \"kubernetes.io/projected/19ea2eab-3e12-4193-8f63-aa7872b284c2-kube-api-access-vzwjn\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.853074 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.852895 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/19ea2eab-3e12-4193-8f63-aa7872b284c2-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.853074 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.852905 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/19ea2eab-3e12-4193-8f63-aa7872b284c2-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:41.853074 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:41.852913 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbc7l\" (UniqueName: \"kubernetes.io/projected/f0c5e6d3-f198-4daa-891f-6af1994e5d89-kube-api-access-dbc7l\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:42.174708 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.174664 6400 generic.go:356] "Generic (PLEG): container finished" podID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerID="bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16" exitCode=0 Oct 30 17:16:42.174936 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.174722 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" Oct 30 17:16:42.174936 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.174764 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" event={"ID":"19ea2eab-3e12-4193-8f63-aa7872b284c2","Type":"ContainerDied","Data":"bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16"} Oct 30 17:16:42.174936 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.174821 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz" event={"ID":"19ea2eab-3e12-4193-8f63-aa7872b284c2","Type":"ContainerDied","Data":"9469783f6d91a1e76ebea7bda6d87f1b2071fa8692ca24296fe1ef6284c1a107"} Oct 30 17:16:42.174936 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.174844 6400 scope.go:122] "RemoveContainer" containerID="bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16" Oct 30 17:16:42.176253 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.176225 6400 generic.go:356] "Generic (PLEG): container finished" podID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerID="6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278" exitCode=0 Oct 30 17:16:42.176373 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.176257 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" event={"ID":"f0c5e6d3-f198-4daa-891f-6af1994e5d89","Type":"ContainerDied","Data":"6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278"} Oct 30 17:16:42.176373 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.176275 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" Oct 30 17:16:42.176373 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.176298 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v" event={"ID":"f0c5e6d3-f198-4daa-891f-6af1994e5d89","Type":"ContainerDied","Data":"35605c825024a5f407156a7899c3c069b1f8bb77b4f2cd0bc96dbc3b6b626e01"} Oct 30 17:16:42.184516 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.184500 6400 scope.go:122] "RemoveContainer" containerID="bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16" Oct 30 17:16:42.184878 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:16:42.184839 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16\": container with ID starting with bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16 not found: ID does not exist" containerID="bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16" Oct 30 17:16:42.184960 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.184886 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16"} err="failed to get container status \"bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16\": rpc error: code = NotFound desc = could not find container \"bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16\": container with ID starting with bc37e062a833fb854612605e519a87ade428ba32bafe7c9c2f964a155e5daf16 not found: ID does not exist" Oct 30 17:16:42.184960 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.184911 6400 scope.go:122] "RemoveContainer" containerID="6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278" Oct 30 17:16:42.193829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.193810 6400 scope.go:122] "RemoveContainer" containerID="6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278" Oct 30 17:16:42.194118 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:16:42.194098 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278\": container with ID starting with 6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278 not found: ID does not exist" containerID="6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278" Oct 30 17:16:42.194196 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.194125 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278"} err="failed to get container status \"6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278\": rpc error: code = NotFound desc = could not find container \"6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278\": container with ID starting with 6d94ed7bd7a7d7c24c1dbe093a510a5de0b9b81bb2be6d9c0ed46f38bae32278 not found: ID does not exist" Oct 30 17:16:42.229831 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.229803 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz"] Oct 30 17:16:42.243111 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.243085 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/premium-w4-263f00-kserve-699df8c945-c4pkz"] Oct 30 17:16:42.272034 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.271999 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v"] Oct 30 17:16:42.275658 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.275631 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-263f00/simulator-w4-263f00-kserve-777999d56d-lkk8v"] Oct 30 17:16:42.966892 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.966859 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" path="/var/lib/kubelet/pods/19ea2eab-3e12-4193-8f63-aa7872b284c2/volumes" Oct 30 17:16:42.967287 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:42.967273 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" path="/var/lib/kubelet/pods/f0c5e6d3-f198-4daa-891f-6af1994e5d89/volumes" Oct 30 17:16:44.419878 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.419842 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws"] Oct 30 17:16:44.420320 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.420083 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" podUID="00a9feb0-d258-4ab0-9d89-82f30f869e6f" containerName="maas-api" containerID="cri-o://6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2" gracePeriod=30 Oct 30 17:16:44.743534 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.743505 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:16:44.798284 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.798251 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00a9feb0-d258-4ab0-9d89-82f30f869e6f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a9feb0-d258-4ab0-9d89-82f30f869e6f-openshift-service-ca\") pod \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " Oct 30 17:16:44.798490 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.798319 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-maas-api-tls\") pod \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " Oct 30 17:16:44.798490 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.798415 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-metrics-certs\") pod \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " Oct 30 17:16:44.798490 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.798473 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00a9feb0-d258-4ab0-9d89-82f30f869e6f-kube-api-access-fvtxn\" (UniqueName: \"kubernetes.io/projected/00a9feb0-d258-4ab0-9d89-82f30f869e6f-kube-api-access-fvtxn\") pod \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\" (UID: \"00a9feb0-d258-4ab0-9d89-82f30f869e6f\") " Oct 30 17:16:44.799133 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.799033 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00a9feb0-d258-4ab0-9d89-82f30f869e6f-openshift-service-ca" pod "00a9feb0-d258-4ab0-9d89-82f30f869e6f" (UID: "00a9feb0-d258-4ab0-9d89-82f30f869e6f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:16:44.799430 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.799388 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/00a9feb0-d258-4ab0-9d89-82f30f869e6f-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:44.802149 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.802122 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00a9feb0-d258-4ab0-9d89-82f30f869e6f-kube-api-access-fvtxn" pod "00a9feb0-d258-4ab0-9d89-82f30f869e6f" (UID: "00a9feb0-d258-4ab0-9d89-82f30f869e6f"). InnerVolumeSpecName "kube-api-access-fvtxn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:16:44.802376 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.802338 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-metrics-certs" pod "00a9feb0-d258-4ab0-9d89-82f30f869e6f" (UID: "00a9feb0-d258-4ab0-9d89-82f30f869e6f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:44.802682 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.802650 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-maas-api-tls" pod "00a9feb0-d258-4ab0-9d89-82f30f869e6f" (UID: "00a9feb0-d258-4ab0-9d89-82f30f869e6f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:16:44.900709 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.900670 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:44.900709 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.900705 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvtxn\" (UniqueName: \"kubernetes.io/projected/00a9feb0-d258-4ab0-9d89-82f30f869e6f-kube-api-access-fvtxn\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:44.900709 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:44.900717 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/00a9feb0-d258-4ab0-9d89-82f30f869e6f-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:16:45.197341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:45.197304 6400 generic.go:356] "Generic (PLEG): container finished" podID="00a9feb0-d258-4ab0-9d89-82f30f869e6f" containerID="6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2" exitCode=0 Oct 30 17:16:45.197560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:45.197382 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" Oct 30 17:16:45.197560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:45.197389 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" event={"ID":"00a9feb0-d258-4ab0-9d89-82f30f869e6f","Type":"ContainerDied","Data":"6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2"} Oct 30 17:16:45.197560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:45.197446 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws" event={"ID":"00a9feb0-d258-4ab0-9d89-82f30f869e6f","Type":"ContainerDied","Data":"92f6c853d745c68fc8396df0e04cea258f2b27d7e17e8cf271c28ad4730cafcb"} Oct 30 17:16:45.197560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:45.197468 6400 scope.go:122] "RemoveContainer" containerID="6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2" Oct 30 17:16:45.207223 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:45.207207 6400 scope.go:122] "RemoveContainer" containerID="6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2" Oct 30 17:16:45.207472 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:16:45.207452 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2\": container with ID starting with 6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2 not found: ID does not exist" containerID="6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2" Oct 30 17:16:45.207547 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:45.207484 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2"} err="failed to get container status \"6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2\": rpc error: code = NotFound desc = could not find container \"6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2\": container with ID starting with 6c18bec8b206f39c055cf5a9093c188f4a17a593ef66900738071039ab6342e2 not found: ID does not exist" Oct 30 17:16:45.252135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:45.252095 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws"] Oct 30 17:16:45.265458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:45.265425 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-ae5afd-65787cd8ff-j6mws"] Oct 30 17:16:46.967855 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:46.967811 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00a9feb0-d258-4ab0-9d89-82f30f869e6f" path="/var/lib/kubelet/pods/00a9feb0-d258-4ab0-9d89-82f30f869e6f/volumes" Oct 30 17:16:54.606730 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:16:54.606691 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:16:57.893213 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:57.893170 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8"] Oct 30 17:16:57.907763 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:16:57.907727 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf"] Oct 30 17:17:05.799252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:05.799217 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2"] Oct 30 17:17:05.800247 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:05.800213 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" containerID="cri-o://7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8" gracePeriod=30 Oct 30 17:17:05.946878 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:05.946842 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w"] Oct 30 17:17:05.947083 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:05.947060 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" podUID="89164063-6e0d-4dc3-9e26-30f9c0557c05" containerName="maas-api" containerID="cri-o://99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca" gracePeriod=30 Oct 30 17:17:06.199773 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.199439 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 17:17:06.268562 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.268528 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-metrics-certs\" (UniqueName: \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-metrics-certs\") pod \"89164063-6e0d-4dc3-9e26-30f9c0557c05\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " Oct 30 17:17:06.268775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.268691 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89164063-6e0d-4dc3-9e26-30f9c0557c05-kube-api-access-6m862\" (UniqueName: \"kubernetes.io/projected/89164063-6e0d-4dc3-9e26-30f9c0557c05-kube-api-access-6m862\") pod \"89164063-6e0d-4dc3-9e26-30f9c0557c05\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " Oct 30 17:17:06.268775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.268750 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-maas-api-tls\") pod \"89164063-6e0d-4dc3-9e26-30f9c0557c05\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " Oct 30 17:17:06.268920 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.268838 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89164063-6e0d-4dc3-9e26-30f9c0557c05-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89164063-6e0d-4dc3-9e26-30f9c0557c05-openshift-service-ca\") pod \"89164063-6e0d-4dc3-9e26-30f9c0557c05\" (UID: \"89164063-6e0d-4dc3-9e26-30f9c0557c05\") " Oct 30 17:17:06.269358 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.269323 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89164063-6e0d-4dc3-9e26-30f9c0557c05-openshift-service-ca" pod "89164063-6e0d-4dc3-9e26-30f9c0557c05" (UID: "89164063-6e0d-4dc3-9e26-30f9c0557c05"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:06.270997 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.270967 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-metrics-certs" pod "89164063-6e0d-4dc3-9e26-30f9c0557c05" (UID: "89164063-6e0d-4dc3-9e26-30f9c0557c05"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:06.271117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.271055 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89164063-6e0d-4dc3-9e26-30f9c0557c05-kube-api-access-6m862" pod "89164063-6e0d-4dc3-9e26-30f9c0557c05" (UID: "89164063-6e0d-4dc3-9e26-30f9c0557c05"). InnerVolumeSpecName "kube-api-access-6m862". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:06.271117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.271094 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-maas-api-tls" pod "89164063-6e0d-4dc3-9e26-30f9c0557c05" (UID: "89164063-6e0d-4dc3-9e26-30f9c0557c05"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:06.321523 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.321445 6400 generic.go:356] "Generic (PLEG): container finished" podID="89164063-6e0d-4dc3-9e26-30f9c0557c05" containerID="99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca" exitCode=0 Oct 30 17:17:06.321523 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.321497 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" Oct 30 17:17:06.321740 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.321522 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" event={"ID":"89164063-6e0d-4dc3-9e26-30f9c0557c05","Type":"ContainerDied","Data":"99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca"} Oct 30 17:17:06.321740 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.321579 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w" event={"ID":"89164063-6e0d-4dc3-9e26-30f9c0557c05","Type":"ContainerDied","Data":"db8492758a4729d6134d44a0ac6d04e9544dc3e00c5d6f0f049555260ef604b9"} Oct 30 17:17:06.321740 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.321630 6400 scope.go:122] "RemoveContainer" containerID="99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca" Oct 30 17:17:06.332497 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.332480 6400 scope.go:122] "RemoveContainer" containerID="99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca" Oct 30 17:17:06.332805 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:06.332786 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca\": container with ID starting with 99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca not found: ID does not exist" containerID="99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca" Oct 30 17:17:06.332883 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.332813 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca"} err="failed to get container status \"99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca\": rpc error: code = NotFound desc = could not find container \"99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca\": container with ID starting with 99eb9972646b170b1baa1ae4cef7689ec013884b0795b69f18d5bbf2bd27f7ca not found: ID does not exist" Oct 30 17:17:06.350585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.350557 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g"] Oct 30 17:17:06.350823 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.350799 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" podUID="eb97e4c4-3a72-4ee0-b98b-4b8e53107bea" containerName="maas-api" containerID="cri-o://0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089" gracePeriod=30 Oct 30 17:17:06.370199 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.370173 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6m862\" (UniqueName: \"kubernetes.io/projected/89164063-6e0d-4dc3-9e26-30f9c0557c05-kube-api-access-6m862\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:06.370325 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.370206 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:06.370325 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.370222 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89164063-6e0d-4dc3-9e26-30f9c0557c05-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:06.370325 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.370235 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89164063-6e0d-4dc3-9e26-30f9c0557c05-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:06.385812 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.385778 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w"] Oct 30 17:17:06.389515 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.389489 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-c8db42-687485b5-bps9w"] Oct 30 17:17:06.621063 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.621037 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 17:17:06.673506 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.673469 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-openshift-service-ca\") pod \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " Oct 30 17:17:06.673756 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.673556 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-kube-api-access-xqwwg\" (UniqueName: \"kubernetes.io/projected/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-kube-api-access-xqwwg\") pod \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " Oct 30 17:17:06.673756 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.673684 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-maas-api-tls\") pod \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " Oct 30 17:17:06.673756 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.673727 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-metrics-certs\") pod \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\" (UID: \"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea\") " Oct 30 17:17:06.675037 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.675002 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-openshift-service-ca" pod "eb97e4c4-3a72-4ee0-b98b-4b8e53107bea" (UID: "eb97e4c4-3a72-4ee0-b98b-4b8e53107bea"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:06.676920 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.676888 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-kube-api-access-xqwwg" pod "eb97e4c4-3a72-4ee0-b98b-4b8e53107bea" (UID: "eb97e4c4-3a72-4ee0-b98b-4b8e53107bea"). InnerVolumeSpecName "kube-api-access-xqwwg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:06.677048 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.676971 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-metrics-certs" pod "eb97e4c4-3a72-4ee0-b98b-4b8e53107bea" (UID: "eb97e4c4-3a72-4ee0-b98b-4b8e53107bea"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:06.677159 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.677138 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-maas-api-tls" pod "eb97e4c4-3a72-4ee0-b98b-4b8e53107bea" (UID: "eb97e4c4-3a72-4ee0-b98b-4b8e53107bea"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:06.774679 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.774644 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:06.774679 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.774677 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xqwwg\" (UniqueName: \"kubernetes.io/projected/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-kube-api-access-xqwwg\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:06.774897 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.774691 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:06.774897 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.774704 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:06.967587 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:06.967555 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89164063-6e0d-4dc3-9e26-30f9c0557c05" path="/var/lib/kubelet/pods/89164063-6e0d-4dc3-9e26-30f9c0557c05/volumes" Oct 30 17:17:07.330516 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:07.330481 6400 generic.go:356] "Generic (PLEG): container finished" podID="eb97e4c4-3a72-4ee0-b98b-4b8e53107bea" containerID="0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089" exitCode=0 Oct 30 17:17:07.330741 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:07.330551 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" event={"ID":"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea","Type":"ContainerDied","Data":"0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089"} Oct 30 17:17:07.330741 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:07.330590 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" event={"ID":"eb97e4c4-3a72-4ee0-b98b-4b8e53107bea","Type":"ContainerDied","Data":"46f8bb52fd653b4a3eab91f7b9b197cd41d61d55a97a84436a3af8b0fa5b0813"} Oct 30 17:17:07.330741 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:07.330551 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g" Oct 30 17:17:07.330741 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:07.330642 6400 scope.go:122] "RemoveContainer" containerID="0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089" Oct 30 17:17:07.340393 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:07.340371 6400 scope.go:122] "RemoveContainer" containerID="0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089" Oct 30 17:17:07.340673 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:07.340654 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089\": container with ID starting with 0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089 not found: ID does not exist" containerID="0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089" Oct 30 17:17:07.340745 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:07.340692 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089"} err="failed to get container status \"0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089\": rpc error: code = NotFound desc = could not find container \"0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089\": container with ID starting with 0e552d1588222ee9b15827e6ca8c119dc67dc0d7e17f021917f7ab5e87b30089 not found: ID does not exist" Oct 30 17:17:07.386406 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:07.386373 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g"] Oct 30 17:17:07.399359 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:07.399329 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w0-bfecdd-65cdc5fbb9-n6z8g"] Oct 30 17:17:08.967465 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:08.967431 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb97e4c4-3a72-4ee0-b98b-4b8e53107bea" path="/var/lib/kubelet/pods/eb97e4c4-3a72-4ee0-b98b-4b8e53107bea/volumes" Oct 30 17:17:11.061199 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.061172 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 17:17:11.123001 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.122911 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-envoy\") pod \"c99286a4-40e3-405c-9e7c-4ca6851357da\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " Oct 30 17:17:11.123001 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.122963 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-certs\") pod \"c99286a4-40e3-405c-9e7c-4ca6851357da\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " Oct 30 17:17:11.123252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.123005 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c99286a4-40e3-405c-9e7c-4ca6851357da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99286a4-40e3-405c-9e7c-4ca6851357da-istiod-ca-cert\") pod \"c99286a4-40e3-405c-9e7c-4ca6851357da\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " Oct 30 17:17:11.123252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.123079 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-podinfo\") pod \"c99286a4-40e3-405c-9e7c-4ca6851357da\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " Oct 30 17:17:11.123252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.123132 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-data\") pod \"c99286a4-40e3-405c-9e7c-4ca6851357da\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " Oct 30 17:17:11.123252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.123183 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-token\" (UniqueName: \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-token\") pod \"c99286a4-40e3-405c-9e7c-4ca6851357da\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " Oct 30 17:17:11.123252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.123228 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-credential-socket\") pod \"c99286a4-40e3-405c-9e7c-4ca6851357da\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " Oct 30 17:17:11.123534 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.123272 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-socket\") pod \"c99286a4-40e3-405c-9e7c-4ca6851357da\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " Oct 30 17:17:11.123534 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.123311 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-kube-api-access-9lb6r\" (UniqueName: \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-kube-api-access-9lb6r\") pod \"c99286a4-40e3-405c-9e7c-4ca6851357da\" (UID: \"c99286a4-40e3-405c-9e7c-4ca6851357da\") " Oct 30 17:17:11.123678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.123636 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-data" pod "c99286a4-40e3-405c-9e7c-4ca6851357da" (UID: "c99286a4-40e3-405c-9e7c-4ca6851357da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:11.124064 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.123875 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-credential-socket" pod "c99286a4-40e3-405c-9e7c-4ca6851357da" (UID: "c99286a4-40e3-405c-9e7c-4ca6851357da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:11.124064 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.123996 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-certs" pod "c99286a4-40e3-405c-9e7c-4ca6851357da" (UID: "c99286a4-40e3-405c-9e7c-4ca6851357da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:11.124251 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.124073 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-socket" pod "c99286a4-40e3-405c-9e7c-4ca6851357da" (UID: "c99286a4-40e3-405c-9e7c-4ca6851357da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:11.124352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.124306 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c99286a4-40e3-405c-9e7c-4ca6851357da-istiod-ca-cert" pod "c99286a4-40e3-405c-9e7c-4ca6851357da" (UID: "c99286a4-40e3-405c-9e7c-4ca6851357da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:11.124805 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.124785 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.124982 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.124967 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.125078 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.125063 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.125167 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.125154 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.125252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.125240 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c99286a4-40e3-405c-9e7c-4ca6851357da-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.127393 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.127358 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-envoy" pod "c99286a4-40e3-405c-9e7c-4ca6851357da" (UID: "c99286a4-40e3-405c-9e7c-4ca6851357da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:11.127501 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.127436 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-token" pod "c99286a4-40e3-405c-9e7c-4ca6851357da" (UID: "c99286a4-40e3-405c-9e7c-4ca6851357da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:11.130121 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.130096 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-kube-api-access-9lb6r" pod "c99286a4-40e3-405c-9e7c-4ca6851357da" (UID: "c99286a4-40e3-405c-9e7c-4ca6851357da"). InnerVolumeSpecName "kube-api-access-9lb6r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:11.130423 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.130403 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-podinfo" pod "c99286a4-40e3-405c-9e7c-4ca6851357da" (UID: "c99286a4-40e3-405c-9e7c-4ca6851357da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:11.226389 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.226353 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.226389 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.226385 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.226663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.226396 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.226663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.226419 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lb6r\" (UniqueName: \"kubernetes.io/projected/c99286a4-40e3-405c-9e7c-4ca6851357da-kube-api-access-9lb6r\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:11.355581 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.355542 6400 generic.go:356] "Generic (PLEG): container finished" podID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerID="7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8" exitCode=0 Oct 30 17:17:11.355809 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.355610 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" Oct 30 17:17:11.355809 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.355634 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" event={"ID":"c99286a4-40e3-405c-9e7c-4ca6851357da","Type":"ContainerDied","Data":"7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8"} Oct 30 17:17:11.355809 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.355688 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2" event={"ID":"c99286a4-40e3-405c-9e7c-4ca6851357da","Type":"ContainerDied","Data":"08e8493eacdfa64a94a802b23fbd723106d9106cee8e7f63fd2df0a755e1d90b"} Oct 30 17:17:11.355809 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.355716 6400 scope.go:122] "RemoveContainer" containerID="7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8" Oct 30 17:17:11.365884 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.365866 6400 scope.go:122] "RemoveContainer" containerID="7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8" Oct 30 17:17:11.366181 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:11.366160 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8\": container with ID starting with 7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8 not found: ID does not exist" containerID="7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8" Oct 30 17:17:11.366231 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.366191 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8"} err="failed to get container status \"7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8\": rpc error: code = NotFound desc = could not find container \"7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8\": container with ID starting with 7d2702833f68978d0e403193bda4727fd98fb3dbc3246849cfa5c36c7ba5d2b8 not found: ID does not exist" Oct 30 17:17:11.410294 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.410217 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2"] Oct 30 17:17:11.423437 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:11.423405 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-263f00-openshift-default-b969d55bc-77lv2"] Oct 30 17:17:12.916806 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:12.916767 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" containerID="cri-o://20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f" gracePeriod=45 Oct 30 17:17:12.928706 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:12.928673 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" containerID="cri-o://89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61" gracePeriod=45 Oct 30 17:17:12.952861 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:12.952830 6400 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c87785a_a2ea_4813_afa9_000a42e7de59.slice/crio-89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7f6209d2_228d_4df9_a185_59381ea836a8.slice/crio-20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c87785a_a2ea_4813_afa9_000a42e7de59.slice/crio-conmon-89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61.scope\": RecentStats: unable to find data in memory cache]" Oct 30 17:17:12.967006 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:12.966978 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" path="/var/lib/kubelet/pods/c99286a4-40e3-405c-9e7c-4ca6851357da/volumes" Oct 30 17:17:13.200864 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.200838 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 17:17:13.226940 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.226912 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 17:17:13.348733 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.348695 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-home\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-home\") pod \"0c87785a-a2ea-4813-afa9-000a42e7de59\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " Oct 30 17:17:13.348910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.348747 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f6209d2-228d-4df9-a185-59381ea836a8-tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6209d2-228d-4df9-a185-59381ea836a8-tls-certs\") pod \"7f6209d2-228d-4df9-a185-59381ea836a8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " Oct 30 17:17:13.348910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.348766 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c87785a-a2ea-4813-afa9-000a42e7de59-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c87785a-a2ea-4813-afa9-000a42e7de59-tls-certs\") pod \"0c87785a-a2ea-4813-afa9-000a42e7de59\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " Oct 30 17:17:13.348910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.348791 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-dshm\") pod \"7f6209d2-228d-4df9-a185-59381ea836a8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " Oct 30 17:17:13.348910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.348833 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-home\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-home\") pod \"7f6209d2-228d-4df9-a185-59381ea836a8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " Oct 30 17:17:13.349134 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.348914 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-model-cache\") pod \"0c87785a-a2ea-4813-afa9-000a42e7de59\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " Oct 30 17:17:13.349134 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.348952 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-tmp-dir\") pod \"7f6209d2-228d-4df9-a185-59381ea836a8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " Oct 30 17:17:13.349134 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.348981 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f6209d2-228d-4df9-a185-59381ea836a8-kube-api-access-6bc8b\" (UniqueName: \"kubernetes.io/projected/7f6209d2-228d-4df9-a185-59381ea836a8-kube-api-access-6bc8b\") pod \"7f6209d2-228d-4df9-a185-59381ea836a8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " Oct 30 17:17:13.349134 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.349014 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-tmp-dir\") pod \"0c87785a-a2ea-4813-afa9-000a42e7de59\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " Oct 30 17:17:13.349134 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.349024 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-home" pod "0c87785a-a2ea-4813-afa9-000a42e7de59" (UID: "0c87785a-a2ea-4813-afa9-000a42e7de59"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:13.349134 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.349066 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-model-cache\") pod \"7f6209d2-228d-4df9-a185-59381ea836a8\" (UID: \"7f6209d2-228d-4df9-a185-59381ea836a8\") " Oct 30 17:17:13.349134 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.349117 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-dshm\") pod \"0c87785a-a2ea-4813-afa9-000a42e7de59\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " Oct 30 17:17:13.349579 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.349146 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c87785a-a2ea-4813-afa9-000a42e7de59-kube-api-access-kzv42\" (UniqueName: \"kubernetes.io/projected/0c87785a-a2ea-4813-afa9-000a42e7de59-kube-api-access-kzv42\") pod \"0c87785a-a2ea-4813-afa9-000a42e7de59\" (UID: \"0c87785a-a2ea-4813-afa9-000a42e7de59\") " Oct 30 17:17:13.349579 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.349207 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-home" pod "7f6209d2-228d-4df9-a185-59381ea836a8" (UID: "7f6209d2-228d-4df9-a185-59381ea836a8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:13.349579 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.349345 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-model-cache" pod "7f6209d2-228d-4df9-a185-59381ea836a8" (UID: "7f6209d2-228d-4df9-a185-59381ea836a8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:13.349579 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.349436 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-tmp-dir" pod "7f6209d2-228d-4df9-a185-59381ea836a8" (UID: "7f6209d2-228d-4df9-a185-59381ea836a8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:13.349579 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.349444 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-model-cache" pod "0c87785a-a2ea-4813-afa9-000a42e7de59" (UID: "0c87785a-a2ea-4813-afa9-000a42e7de59"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:13.350295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.350099 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.350295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.350127 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.350295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.350139 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.350295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.350153 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.350295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.350194 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.350295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.350267 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-tmp-dir" pod "0c87785a-a2ea-4813-afa9-000a42e7de59" (UID: "0c87785a-a2ea-4813-afa9-000a42e7de59"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:13.351783 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.351759 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f6209d2-228d-4df9-a185-59381ea836a8-tls-certs" pod "7f6209d2-228d-4df9-a185-59381ea836a8" (UID: "7f6209d2-228d-4df9-a185-59381ea836a8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:13.351897 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.351769 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-dshm" pod "7f6209d2-228d-4df9-a185-59381ea836a8" (UID: "7f6209d2-228d-4df9-a185-59381ea836a8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:13.351944 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.351895 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c87785a-a2ea-4813-afa9-000a42e7de59-kube-api-access-kzv42" pod "0c87785a-a2ea-4813-afa9-000a42e7de59" (UID: "0c87785a-a2ea-4813-afa9-000a42e7de59"). InnerVolumeSpecName "kube-api-access-kzv42". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:13.351944 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.351933 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-dshm" pod "0c87785a-a2ea-4813-afa9-000a42e7de59" (UID: "0c87785a-a2ea-4813-afa9-000a42e7de59"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:13.352056 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.352042 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c87785a-a2ea-4813-afa9-000a42e7de59-tls-certs" pod "0c87785a-a2ea-4813-afa9-000a42e7de59" (UID: "0c87785a-a2ea-4813-afa9-000a42e7de59"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:13.352150 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.352131 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f6209d2-228d-4df9-a185-59381ea836a8-kube-api-access-6bc8b" pod "7f6209d2-228d-4df9-a185-59381ea836a8" (UID: "7f6209d2-228d-4df9-a185-59381ea836a8"). InnerVolumeSpecName "kube-api-access-6bc8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:13.377347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.377311 6400 generic.go:356] "Generic (PLEG): container finished" podID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerID="89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61" exitCode=0 Oct 30 17:17:13.377499 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.377356 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" Oct 30 17:17:13.377499 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.377393 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" event={"ID":"0c87785a-a2ea-4813-afa9-000a42e7de59","Type":"ContainerDied","Data":"89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61"} Oct 30 17:17:13.377499 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.377437 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf" event={"ID":"0c87785a-a2ea-4813-afa9-000a42e7de59","Type":"ContainerDied","Data":"7ff3226f25be33ecb280102e081811b0f8b3035fe3e6eef0ad8bd69df6e78d34"} Oct 30 17:17:13.377499 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.377455 6400 scope.go:122] "RemoveContainer" containerID="89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61" Oct 30 17:17:13.378723 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.378700 6400 generic.go:356] "Generic (PLEG): container finished" podID="7f6209d2-228d-4df9-a185-59381ea836a8" containerID="20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f" exitCode=0 Oct 30 17:17:13.378841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.378746 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" Oct 30 17:17:13.378841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.378767 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" event={"ID":"7f6209d2-228d-4df9-a185-59381ea836a8","Type":"ContainerDied","Data":"20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f"} Oct 30 17:17:13.378841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.378803 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8" event={"ID":"7f6209d2-228d-4df9-a185-59381ea836a8","Type":"ContainerDied","Data":"aa8bc1f41dfa9ec7f058ce7e11610b9d43fce9fc469ecda77ae52c9aa08d29ce"} Oct 30 17:17:13.386650 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.386636 6400 scope.go:122] "RemoveContainer" containerID="89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61" Oct 30 17:17:13.386985 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:13.386959 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61\": container with ID starting with 89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61 not found: ID does not exist" containerID="89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61" Oct 30 17:17:13.387105 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.386989 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61"} err="failed to get container status \"89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61\": rpc error: code = NotFound desc = could not find container \"89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61\": container with ID starting with 89a625b6fdcd827195d9d0d13486404ee48e85462a8ade099765e1bcca8c4e61 not found: ID does not exist" Oct 30 17:17:13.387105 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.387006 6400 scope.go:122] "RemoveContainer" containerID="20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f" Oct 30 17:17:13.397093 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.397067 6400 scope.go:122] "RemoveContainer" containerID="20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f" Oct 30 17:17:13.397431 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:13.397374 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f\": container with ID starting with 20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f not found: ID does not exist" containerID="20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f" Oct 30 17:17:13.397431 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.397401 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f"} err="failed to get container status \"20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f\": rpc error: code = NotFound desc = could not find container \"20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f\": container with ID starting with 20d2ade02510e48b0ae745278d8cdc0faf7489dc474592ad6c3a993453b1426f not found: ID does not exist" Oct 30 17:17:13.432525 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.432449 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8"] Oct 30 17:17:13.434679 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.434659 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/premium-w1-a8c22d-kserve-865c4784f5-tsgv8"] Oct 30 17:17:13.451182 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.451153 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7f6209d2-228d-4df9-a185-59381ea836a8-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.451182 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.451175 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c87785a-a2ea-4813-afa9-000a42e7de59-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.451182 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.451184 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7f6209d2-228d-4df9-a185-59381ea836a8-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.451376 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.451192 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bc8b\" (UniqueName: \"kubernetes.io/projected/7f6209d2-228d-4df9-a185-59381ea836a8-kube-api-access-6bc8b\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.451376 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.451201 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.451376 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.451211 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c87785a-a2ea-4813-afa9-000a42e7de59-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.451376 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.451219 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzv42\" (UniqueName: \"kubernetes.io/projected/0c87785a-a2ea-4813-afa9-000a42e7de59-kube-api-access-kzv42\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:13.471801 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.471772 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf"] Oct 30 17:17:13.473706 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:13.473686 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-a8c22d/simulator-w1-a8c22d-kserve-5cb49b6754-4zvxf"] Oct 30 17:17:14.966032 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:14.965991 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" path="/var/lib/kubelet/pods/0c87785a-a2ea-4813-afa9-000a42e7de59/volumes" Oct 30 17:17:14.966536 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:14.966517 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" path="/var/lib/kubelet/pods/7f6209d2-228d-4df9-a185-59381ea836a8/volumes" Oct 30 17:17:16.423952 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.423914 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk"] Oct 30 17:17:16.424817 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424790 6400 cpu_manager.go:414] "containerMap: removing container" podUID="75cea706-1357-4bf0-a987-6323bbf35317" containerName="cleanup" Oct 30 17:17:16.424817 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424818 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cea706-1357-4bf0-a987-6323bbf35317" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424842 6400 cpu_manager.go:414] "containerMap: removing container" podUID="75cea706-1357-4bf0-a987-6323bbf35317" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424850 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cea706-1357-4bf0-a987-6323bbf35317" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424887 6400 cpu_manager.go:414] "containerMap: removing container" podUID="510145d1-4707-4356-80de-09cf7da6540f" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424895 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510145d1-4707-4356-80de-09cf7da6540f" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424906 6400 cpu_manager.go:414] "containerMap: removing container" podUID="510145d1-4707-4356-80de-09cf7da6540f" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424913 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510145d1-4707-4356-80de-09cf7da6540f" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424927 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerName="istio-proxy" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424939 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerName="istio-proxy" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424948 6400 cpu_manager.go:414] "containerMap: removing container" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerName="istio-proxy" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424956 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerName="istio-proxy" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424973 6400 cpu_manager.go:414] "containerMap: removing container" podUID="75cea706-1357-4bf0-a987-6323bbf35317" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424980 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75cea706-1357-4bf0-a987-6323bbf35317" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424990 6400 cpu_manager.go:414] "containerMap: removing container" podUID="91c62170-6668-4436-b794-10170604b7d0" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.424997 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c62170-6668-4436-b794-10170604b7d0" containerName="cleanup" Oct 30 17:17:16.425010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425018 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerName="cleanup" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425026 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerName="cleanup" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425038 6400 cpu_manager.go:414] "containerMap: removing container" podUID="00a9feb0-d258-4ab0-9d89-82f30f869e6f" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425044 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00a9feb0-d258-4ab0-9d89-82f30f869e6f" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425060 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425068 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425095 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f28e6474-6a1e-4727-b3c4-9ea757a7ea3d" containerName="payload-pre-processing" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425102 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28e6474-6a1e-4727-b3c4-9ea757a7ea3d" containerName="payload-pre-processing" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425118 6400 cpu_manager.go:414] "containerMap: removing container" podUID="510145d1-4707-4356-80de-09cf7da6540f" containerName="cleanup" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425124 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="510145d1-4707-4356-80de-09cf7da6540f" containerName="cleanup" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425136 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a7e415d4-b0b2-40c1-b9fe-beb23f9c0809" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425142 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a7e415d4-b0b2-40c1-b9fe-beb23f9c0809" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425162 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425168 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425184 6400 cpu_manager.go:414] "containerMap: removing container" podUID="52ba393f-b73e-4e8e-b509-12dabb21e851" containerName="payload-processing" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425191 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ba393f-b73e-4e8e-b509-12dabb21e851" containerName="payload-processing" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425207 6400 cpu_manager.go:414] "containerMap: removing container" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425214 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425223 6400 cpu_manager.go:414] "containerMap: removing container" podUID="eb97e4c4-3a72-4ee0-b98b-4b8e53107bea" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425229 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb97e4c4-3a72-4ee0-b98b-4b8e53107bea" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425246 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerName="cleanup" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425254 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerName="cleanup" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425263 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425270 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425293 6400 cpu_manager.go:414] "containerMap: removing container" podUID="bba32efe-e717-4821-996d-9da99af18277" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425301 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bba32efe-e717-4821-996d-9da99af18277" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425310 6400 cpu_manager.go:414] "containerMap: removing container" podUID="8978fa9f-b2c7-4132-a36f-69c9a4a9389c" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425316 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8978fa9f-b2c7-4132-a36f-69c9a4a9389c" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425333 6400 cpu_manager.go:414] "containerMap: removing container" podUID="641a877c-4c86-4e4c-b331-c8f71e8760a1" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425339 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="641a877c-4c86-4e4c-b331-c8f71e8760a1" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425348 6400 cpu_manager.go:414] "containerMap: removing container" podUID="89164063-6e0d-4dc3-9e26-30f9c0557c05" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425361 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89164063-6e0d-4dc3-9e26-30f9c0557c05" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425372 6400 cpu_manager.go:414] "containerMap: removing container" podUID="80ef9893-a9f8-47b8-82cc-872188fbc467" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425379 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80ef9893-a9f8-47b8-82cc-872188fbc467" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425407 6400 cpu_manager.go:414] "containerMap: removing container" podUID="91c62170-6668-4436-b794-10170604b7d0" containerName="cleanup" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425414 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c62170-6668-4436-b794-10170604b7d0" containerName="cleanup" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425424 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425431 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425446 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e32f582c-5e90-4129-aa85-cdf31c99d0ac" containerName="revoke-keys" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425453 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e32f582c-5e90-4129-aa85-cdf31c99d0ac" containerName="revoke-keys" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425466 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425473 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425491 6400 cpu_manager.go:414] "containerMap: removing container" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425498 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425514 6400 cpu_manager.go:414] "containerMap: removing container" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425521 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425726 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a7e415d4-b0b2-40c1-b9fe-beb23f9c0809" containerName="maas-api" Oct 30 17:17:16.425729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425750 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c1fd2b9-f051-4cbe-a305-57d6136e2cf7" containerName="main" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425766 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ba393f-b73e-4e8e-b509-12dabb21e851" containerName="payload-processing" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425781 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c87785a-a2ea-4813-afa9-000a42e7de59" containerName="main" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425797 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb97e4c4-3a72-4ee0-b98b-4b8e53107bea" containerName="maas-api" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425806 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerName="cleanup" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425814 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerName="cleanup" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425829 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0c5e6d3-f198-4daa-891f-6af1994e5d89" containerName="main" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425838 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c62170-6668-4436-b794-10170604b7d0" containerName="cleanup" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425848 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="bba32efe-e717-4821-996d-9da99af18277" containerName="maas-api" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425865 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="8978fa9f-b2c7-4132-a36f-69c9a4a9389c" containerName="maas-api" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425879 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="641a877c-4c86-4e4c-b331-c8f71e8760a1" containerName="maas-api" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425889 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="89164063-6e0d-4dc3-9e26-30f9c0557c05" containerName="maas-api" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425903 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="80ef9893-a9f8-47b8-82cc-872188fbc467" containerName="maas-api" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425919 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c99286a4-40e3-405c-9e7c-4ca6851357da" containerName="istio-proxy" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425927 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e32f582c-5e90-4129-aa85-cdf31c99d0ac" containerName="revoke-keys" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425936 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e2923a4-b7a1-49b9-95b1-1ff146f5f39d" containerName="maas-api" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425951 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f6209d2-228d-4df9-a185-59381ea836a8" containerName="main" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425966 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ea2eab-3e12-4193-8f63-aa7872b284c2" containerName="main" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425981 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cea706-1357-4bf0-a987-6323bbf35317" containerName="cleanup" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425988 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cea706-1357-4bf0-a987-6323bbf35317" containerName="cleanup" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.425998 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="510145d1-4707-4356-80de-09cf7da6540f" containerName="cleanup" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.426012 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="510145d1-4707-4356-80de-09cf7da6540f" containerName="cleanup" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.426028 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1e4f957-c66c-40dc-8560-a6bbd2cd75d1" containerName="istio-proxy" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.426037 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="05edb360-ea8e-4b72-846d-e3ecd317bb37" containerName="istio-proxy" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.426045 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c62170-6668-4436-b794-10170604b7d0" containerName="cleanup" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.426059 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.426067 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="00a9feb0-d258-4ab0-9d89-82f30f869e6f" containerName="maas-api" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.426075 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" Oct 30 17:17:16.427942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.426085 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28e6474-6a1e-4727-b3c4-9ea757a7ea3d" containerName="payload-pre-processing" Oct 30 17:17:16.431804 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.431782 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:16.447869 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.447845 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk"] Oct 30 17:17:16.458496 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.458455 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:17:16.458496 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.458485 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:17:16.583264 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.583228 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:16.583264 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.583265 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ljkf\" (UniqueName: \"kubernetes.io/projected/df1c8a3a-c13e-4e6f-b326-6f7afc988056-kube-api-access-9ljkf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:16.583504 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.583453 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:16.684399 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.684318 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:16.684399 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.684375 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:16.684399 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.684396 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ljkf\" (UniqueName: \"kubernetes.io/projected/df1c8a3a-c13e-4e6f-b326-6f7afc988056-kube-api-access-9ljkf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:16.685135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.685107 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:16.687058 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.687038 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:16.709165 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.709144 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ljkf\" (UniqueName: \"kubernetes.io/projected/df1c8a3a-c13e-4e6f-b326-6f7afc988056-kube-api-access-9ljkf\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:16.745139 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:16.745118 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:17.054297 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:17.054270 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk"] Oct 30 17:17:17.055863 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:17:17.055824 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf1c8a3a_c13e_4e6f_b326_6f7afc988056.slice/crio-20c1670cc5fe5e3a52da5bf0b1b5eef5c50438aeb25cd5c2be7285cc6162ad8d WatchSource:0}: Error finding container 20c1670cc5fe5e3a52da5bf0b1b5eef5c50438aeb25cd5c2be7285cc6162ad8d: Status 404 returned error can't find the container with id 20c1670cc5fe5e3a52da5bf0b1b5eef5c50438aeb25cd5c2be7285cc6162ad8d Oct 30 17:17:17.407253 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:17.407214 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" event={"ID":"df1c8a3a-c13e-4e6f-b326-6f7afc988056","Type":"ContainerStarted","Data":"c0e0560c3f2c0f37781f4a0291bc4a0bd328643abc094b90f23653500c699051"} Oct 30 17:17:17.407444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:17.407275 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" event={"ID":"df1c8a3a-c13e-4e6f-b326-6f7afc988056","Type":"ContainerStarted","Data":"20c1670cc5fe5e3a52da5bf0b1b5eef5c50438aeb25cd5c2be7285cc6162ad8d"} Oct 30 17:17:17.463579 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:17.463514 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" podStartSLOduration=1.46349942 podStartE2EDuration="1.46349942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:17.449784684 +0000 UTC m=+3346.600065672" watchObservedRunningTime="2026-10-30 17:17:17.46349942 +0000 UTC m=+3346.613780407" Oct 30 17:17:18.415251 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:18.415209 6400 generic.go:356] "Generic (PLEG): container finished" podID="df1c8a3a-c13e-4e6f-b326-6f7afc988056" containerID="c0e0560c3f2c0f37781f4a0291bc4a0bd328643abc094b90f23653500c699051" exitCode=0 Oct 30 17:17:18.415445 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:18.415302 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" event={"ID":"df1c8a3a-c13e-4e6f-b326-6f7afc988056","Type":"ContainerDied","Data":"c0e0560c3f2c0f37781f4a0291bc4a0bd328643abc094b90f23653500c699051"} Oct 30 17:17:19.352363 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.352324 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p"] Oct 30 17:17:19.352905 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.352596 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" containerID="cri-o://cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2" gracePeriod=30 Oct 30 17:17:19.551356 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.551329 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:19.716798 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.716712 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-openshift-ca\") pod \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " Oct 30 17:17:19.716798 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.716750 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-xks-ca\") pod \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " Oct 30 17:17:19.717015 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.716855 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/df1c8a3a-c13e-4e6f-b326-6f7afc988056-kube-api-access-9ljkf\" (UniqueName: \"kubernetes.io/projected/df1c8a3a-c13e-4e6f-b326-6f7afc988056-kube-api-access-9ljkf\") pod \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\" (UID: \"df1c8a3a-c13e-4e6f-b326-6f7afc988056\") " Oct 30 17:17:19.717125 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.717099 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-openshift-ca" pod "df1c8a3a-c13e-4e6f-b326-6f7afc988056" (UID: "df1c8a3a-c13e-4e6f-b326-6f7afc988056"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:19.717277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.717261 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:19.719114 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.719092 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df1c8a3a-c13e-4e6f-b326-6f7afc988056-kube-api-access-9ljkf" pod "df1c8a3a-c13e-4e6f-b326-6f7afc988056" (UID: "df1c8a3a-c13e-4e6f-b326-6f7afc988056"). InnerVolumeSpecName "kube-api-access-9ljkf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:19.719180 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.719156 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-xks-ca" pod "df1c8a3a-c13e-4e6f-b326-6f7afc988056" (UID: "df1c8a3a-c13e-4e6f-b326-6f7afc988056"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:19.818142 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.818110 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ljkf\" (UniqueName: \"kubernetes.io/projected/df1c8a3a-c13e-4e6f-b326-6f7afc988056-kube-api-access-9ljkf\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:19.818142 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:19.818142 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/df1c8a3a-c13e-4e6f-b326-6f7afc988056-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:20.334988 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:20.334957 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d"] Oct 30 17:17:20.350860 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:20.350827 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z"] Oct 30 17:17:20.362996 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:20.362966 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8"] Oct 30 17:17:20.428729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:20.428689 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" Oct 30 17:17:20.428729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:20.428700 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk" event={"ID":"df1c8a3a-c13e-4e6f-b326-6f7afc988056","Type":"ContainerDied","Data":"20c1670cc5fe5e3a52da5bf0b1b5eef5c50438aeb25cd5c2be7285cc6162ad8d"} Oct 30 17:17:20.429004 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:20.428748 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="20c1670cc5fe5e3a52da5bf0b1b5eef5c50438aeb25cd5c2be7285cc6162ad8d" Oct 30 17:17:24.603779 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:24.603738 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:17:24.616678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.616655 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 17:17:24.676427 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676334 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-kube-api-access-psmnw\" (UniqueName: \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-kube-api-access-psmnw\") pod \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " Oct 30 17:17:24.676427 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676423 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-certs\") pod \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " Oct 30 17:17:24.676646 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676467 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-token\" (UniqueName: \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-token\") pod \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " Oct 30 17:17:24.676646 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676493 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-data\") pod \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " Oct 30 17:17:24.676646 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676548 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-envoy\") pod \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " Oct 30 17:17:24.676646 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676584 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istiod-ca-cert\") pod \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " Oct 30 17:17:24.676885 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676661 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-credential-socket\") pod \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " Oct 30 17:17:24.676885 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676722 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-podinfo\") pod \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " Oct 30 17:17:24.676885 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676748 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-socket\") pod \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\" (UID: \"d90b88ae-d012-4128-9479-ed9f5e8bbff2\") " Oct 30 17:17:24.676885 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676825 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-certs" pod "d90b88ae-d012-4128-9479-ed9f5e8bbff2" (UID: "d90b88ae-d012-4128-9479-ed9f5e8bbff2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:24.676885 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676876 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-data" pod "d90b88ae-d012-4128-9479-ed9f5e8bbff2" (UID: "d90b88ae-d012-4128-9479-ed9f5e8bbff2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:24.677152 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.676976 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-credential-socket" pod "d90b88ae-d012-4128-9479-ed9f5e8bbff2" (UID: "d90b88ae-d012-4128-9479-ed9f5e8bbff2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:24.677152 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.677138 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-socket" pod "d90b88ae-d012-4128-9479-ed9f5e8bbff2" (UID: "d90b88ae-d012-4128-9479-ed9f5e8bbff2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:24.677247 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.677164 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istiod-ca-cert" pod "d90b88ae-d012-4128-9479-ed9f5e8bbff2" (UID: "d90b88ae-d012-4128-9479-ed9f5e8bbff2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:24.677247 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.677223 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:24.677324 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.677245 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:24.677324 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.677260 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:24.679128 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.679101 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-podinfo" pod "d90b88ae-d012-4128-9479-ed9f5e8bbff2" (UID: "d90b88ae-d012-4128-9479-ed9f5e8bbff2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:24.679128 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.679111 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-envoy" pod "d90b88ae-d012-4128-9479-ed9f5e8bbff2" (UID: "d90b88ae-d012-4128-9479-ed9f5e8bbff2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:24.679420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.679401 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-kube-api-access-psmnw" pod "d90b88ae-d012-4128-9479-ed9f5e8bbff2" (UID: "d90b88ae-d012-4128-9479-ed9f5e8bbff2"). InnerVolumeSpecName "kube-api-access-psmnw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:24.679491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.679430 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-token" pod "d90b88ae-d012-4128-9479-ed9f5e8bbff2" (UID: "d90b88ae-d012-4128-9479-ed9f5e8bbff2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:24.778549 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.778502 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:24.778549 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.778541 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:24.778549 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.778550 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:24.778549 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.778560 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d90b88ae-d012-4128-9479-ed9f5e8bbff2-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:24.778549 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.778568 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d90b88ae-d012-4128-9479-ed9f5e8bbff2-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:24.778899 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:24.778578 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psmnw\" (UniqueName: \"kubernetes.io/projected/d90b88ae-d012-4128-9479-ed9f5e8bbff2-kube-api-access-psmnw\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:25.457166 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:25.457129 6400 generic.go:356] "Generic (PLEG): container finished" podID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerID="cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2" exitCode=0 Oct 30 17:17:25.457383 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:25.457183 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" Oct 30 17:17:25.457383 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:25.457185 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" event={"ID":"d90b88ae-d012-4128-9479-ed9f5e8bbff2","Type":"ContainerDied","Data":"cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2"} Oct 30 17:17:25.457383 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:25.457320 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p" event={"ID":"d90b88ae-d012-4128-9479-ed9f5e8bbff2","Type":"ContainerDied","Data":"a89a680b355073a95ebc05ab74c17ed281f3df8a050d2a6d610ddf085d24503e"} Oct 30 17:17:25.457383 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:25.457348 6400 scope.go:122] "RemoveContainer" containerID="cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2" Oct 30 17:17:25.467198 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:25.467181 6400 scope.go:122] "RemoveContainer" containerID="cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2" Oct 30 17:17:25.467512 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:25.467486 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2\": container with ID starting with cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2 not found: ID does not exist" containerID="cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2" Oct 30 17:17:25.467663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:25.467520 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2"} err="failed to get container status \"cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2\": rpc error: code = NotFound desc = could not find container \"cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2\": container with ID starting with cd1043145109c3d7e0178486ec58e3cd3ef4b323b0ea8dfedbbd8796055748e2 not found: ID does not exist" Oct 30 17:17:25.525350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:25.525317 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p"] Oct 30 17:17:25.538678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:25.538652 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-0b2348-openshift-default-85679bc47c-gwd9p"] Oct 30 17:17:26.965874 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:26.965837 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" path="/var/lib/kubelet/pods/d90b88ae-d012-4128-9479-ed9f5e8bbff2/volumes" Oct 30 17:17:30.860295 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.860262 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt"] Oct 30 17:17:30.860940 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.860918 6400 cpu_manager.go:414] "containerMap: removing container" podUID="91c62170-6668-4436-b794-10170604b7d0" containerName="cleanup" Oct 30 17:17:30.861004 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.860944 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91c62170-6668-4436-b794-10170604b7d0" containerName="cleanup" Oct 30 17:17:30.861004 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.860968 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerName="cleanup" Oct 30 17:17:30.861004 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.860976 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerName="cleanup" Oct 30 17:17:30.861004 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.860991 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" Oct 30 17:17:30.861004 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.860998 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a964e5-301d-4f97-b692-f2c9797276c4" containerName="istio-proxy" Oct 30 17:17:30.861154 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.861027 6400 cpu_manager.go:414] "containerMap: removing container" podUID="df1c8a3a-c13e-4e6f-b326-6f7afc988056" containerName="revoke-keys" Oct 30 17:17:30.861154 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.861035 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="df1c8a3a-c13e-4e6f-b326-6f7afc988056" containerName="revoke-keys" Oct 30 17:17:30.861154 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.861065 6400 cpu_manager.go:414] "containerMap: removing container" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" Oct 30 17:17:30.861154 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.861072 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" Oct 30 17:17:30.861272 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.861176 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d90b88ae-d012-4128-9479-ed9f5e8bbff2" containerName="istio-proxy" Oct 30 17:17:30.861272 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.861189 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="75cea706-1357-4bf0-a987-6323bbf35317" containerName="cleanup" Oct 30 17:17:30.861272 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.861200 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="510145d1-4707-4356-80de-09cf7da6540f" containerName="cleanup" Oct 30 17:17:30.861272 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.861210 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="91c62170-6668-4436-b794-10170604b7d0" containerName="cleanup" Oct 30 17:17:30.861272 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.861223 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="df1c8a3a-c13e-4e6f-b326-6f7afc988056" containerName="revoke-keys" Oct 30 17:17:30.861272 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.861234 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9267b55-cd1d-4d80-8c38-81fbdf673d64" containerName="cleanup" Oct 30 17:17:30.866347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.866324 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:30.880858 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.880832 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb"] Oct 30 17:17:30.881319 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.881270 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" podUID="a4653719-1cc9-451a-8634-fa8924d6ac41" containerName="maas-api" containerID="cri-o://faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680" gracePeriod=30 Oct 30 17:17:30.893632 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.893203 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:17:30.895841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.895815 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:17:30.900821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.900791 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt"] Oct 30 17:17:30.942782 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.942756 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fczjz\" (UniqueName: \"kubernetes.io/projected/ce8ad726-6306-482d-9b5b-90c853b0d222-kube-api-access-fczjz\") pod \"maas-api-revoke-sub-af8274de-dldxt\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:30.942942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.942802 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af8274de-dldxt\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:30.942987 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:30.942945 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af8274de-dldxt\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:31.043774 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.043738 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fczjz\" (UniqueName: \"kubernetes.io/projected/ce8ad726-6306-482d-9b5b-90c853b0d222-kube-api-access-fczjz\") pod \"maas-api-revoke-sub-af8274de-dldxt\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:31.043932 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.043807 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af8274de-dldxt\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:31.043932 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.043891 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af8274de-dldxt\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:31.044630 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.044591 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-af8274de-dldxt\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:31.058755 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.058731 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:17:31.067345 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.066987 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-xks-ca\") pod \"maas-api-revoke-sub-af8274de-dldxt\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:31.073949 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.073573 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fczjz\" (UniqueName: \"kubernetes.io/projected/ce8ad726-6306-482d-9b5b-90c853b0d222-kube-api-access-fczjz\") pod \"maas-api-revoke-sub-af8274de-dldxt\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:31.169524 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.169495 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 17:17:31.195019 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.194992 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:17:31.201085 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.201069 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:31.245619 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.245571 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4653719-1cc9-451a-8634-fa8924d6ac41-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4653719-1cc9-451a-8634-fa8924d6ac41-openshift-service-ca\") pod \"a4653719-1cc9-451a-8634-fa8924d6ac41\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " Oct 30 17:17:31.245775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.245666 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-maas-api-tls\") pod \"a4653719-1cc9-451a-8634-fa8924d6ac41\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " Oct 30 17:17:31.245775 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.245713 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4653719-1cc9-451a-8634-fa8924d6ac41-kube-api-access-fg6bf\" (UniqueName: \"kubernetes.io/projected/a4653719-1cc9-451a-8634-fa8924d6ac41-kube-api-access-fg6bf\") pod \"a4653719-1cc9-451a-8634-fa8924d6ac41\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " Oct 30 17:17:31.245858 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.245824 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-metrics-certs\") pod \"a4653719-1cc9-451a-8634-fa8924d6ac41\" (UID: \"a4653719-1cc9-451a-8634-fa8924d6ac41\") " Oct 30 17:17:31.245988 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.245968 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4653719-1cc9-451a-8634-fa8924d6ac41-openshift-service-ca" pod "a4653719-1cc9-451a-8634-fa8924d6ac41" (UID: "a4653719-1cc9-451a-8634-fa8924d6ac41"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:31.246269 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.246248 6400 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4653719-1cc9-451a-8634-fa8924d6ac41-openshift-service-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.248127 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.248092 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-maas-api-tls" pod "a4653719-1cc9-451a-8634-fa8924d6ac41" (UID: "a4653719-1cc9-451a-8634-fa8924d6ac41"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:31.248127 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.248119 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4653719-1cc9-451a-8634-fa8924d6ac41-kube-api-access-fg6bf" pod "a4653719-1cc9-451a-8634-fa8924d6ac41" (UID: "a4653719-1cc9-451a-8634-fa8924d6ac41"). InnerVolumeSpecName "kube-api-access-fg6bf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:31.248640 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.248582 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-metrics-certs" pod "a4653719-1cc9-451a-8634-fa8924d6ac41" (UID: "a4653719-1cc9-451a-8634-fa8924d6ac41"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:31.347713 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.347673 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-maas-api-tls\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.347713 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.347710 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg6bf\" (UniqueName: \"kubernetes.io/projected/a4653719-1cc9-451a-8634-fa8924d6ac41-kube-api-access-fg6bf\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.347957 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.347727 6400 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a4653719-1cc9-451a-8634-fa8924d6ac41-metrics-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:31.495528 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.495498 6400 generic.go:356] "Generic (PLEG): container finished" podID="a4653719-1cc9-451a-8634-fa8924d6ac41" containerID="faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680" exitCode=0 Oct 30 17:17:31.495727 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.495628 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" Oct 30 17:17:31.495727 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.495635 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" event={"ID":"a4653719-1cc9-451a-8634-fa8924d6ac41","Type":"ContainerDied","Data":"faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680"} Oct 30 17:17:31.495727 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.495703 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb" event={"ID":"a4653719-1cc9-451a-8634-fa8924d6ac41","Type":"ContainerDied","Data":"fba88f9ffe60caed3161dabff60adfc69bd10eb42efd86a2f9359d42ce231307"} Oct 30 17:17:31.495872 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.495726 6400 scope.go:122] "RemoveContainer" containerID="faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680" Oct 30 17:17:31.502918 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.502893 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt"] Oct 30 17:17:31.506998 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.506975 6400 scope.go:122] "RemoveContainer" containerID="faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680" Oct 30 17:17:31.507278 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:31.507257 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680\": container with ID starting with faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680 not found: ID does not exist" containerID="faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680" Oct 30 17:17:31.507361 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.507286 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680"} err="failed to get container status \"faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680\": rpc error: code = NotFound desc = could not find container \"faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680\": container with ID starting with faa21d3d9b577457c8e0ee491c86cf3f11d94a7c9875db57a0e687d6779d8680 not found: ID does not exist" Oct 30 17:17:31.508395 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:17:31.508371 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8ad726_6306_482d_9b5b_90c853b0d222.slice/crio-42ff829503ea164186b8a9de0343d9deb242ddf5d8c3885e7a7c4ed8fb8ee758 WatchSource:0}: Error finding container 42ff829503ea164186b8a9de0343d9deb242ddf5d8c3885e7a7c4ed8fb8ee758: Status 404 returned error can't find the container with id 42ff829503ea164186b8a9de0343d9deb242ddf5d8c3885e7a7c4ed8fb8ee758 Oct 30 17:17:31.551435 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.551409 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb"] Oct 30 17:17:31.557655 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.557634 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w0-9e533f-5c9768d979-hqlwb"] Oct 30 17:17:31.568199 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.568172 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk"] Oct 30 17:17:31.570664 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:31.570628 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-9e533f-4c6jk"] Oct 30 17:17:32.505855 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:32.505822 6400 generic.go:356] "Generic (PLEG): container finished" podID="ce8ad726-6306-482d-9b5b-90c853b0d222" containerID="3f05d77819d5c758d27ecd2ca53de71b89ca4b702f68d3de077be5a3df711562" exitCode=0 Oct 30 17:17:32.506362 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:32.505925 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" event={"ID":"ce8ad726-6306-482d-9b5b-90c853b0d222","Type":"ContainerDied","Data":"3f05d77819d5c758d27ecd2ca53de71b89ca4b702f68d3de077be5a3df711562"} Oct 30 17:17:32.506362 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:32.506020 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" event={"ID":"ce8ad726-6306-482d-9b5b-90c853b0d222","Type":"ContainerStarted","Data":"42ff829503ea164186b8a9de0343d9deb242ddf5d8c3885e7a7c4ed8fb8ee758"} Oct 30 17:17:32.967017 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:32.966923 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4653719-1cc9-451a-8634-fa8924d6ac41" path="/var/lib/kubelet/pods/a4653719-1cc9-451a-8634-fa8924d6ac41/volumes" Oct 30 17:17:32.967423 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:32.967401 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="df1c8a3a-c13e-4e6f-b326-6f7afc988056" path="/var/lib/kubelet/pods/df1c8a3a-c13e-4e6f-b326-6f7afc988056/volumes" Oct 30 17:17:33.652551 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:33.652525 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:33.774157 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:33.774080 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ce8ad726-6306-482d-9b5b-90c853b0d222-kube-api-access-fczjz\" (UniqueName: \"kubernetes.io/projected/ce8ad726-6306-482d-9b5b-90c853b0d222-kube-api-access-fczjz\") pod \"ce8ad726-6306-482d-9b5b-90c853b0d222\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " Oct 30 17:17:33.774157 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:33.774119 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-xks-ca\") pod \"ce8ad726-6306-482d-9b5b-90c853b0d222\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " Oct 30 17:17:33.774361 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:33.774172 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-openshift-ca\") pod \"ce8ad726-6306-482d-9b5b-90c853b0d222\" (UID: \"ce8ad726-6306-482d-9b5b-90c853b0d222\") " Oct 30 17:17:33.774672 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:33.774649 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-openshift-ca" pod "ce8ad726-6306-482d-9b5b-90c853b0d222" (UID: "ce8ad726-6306-482d-9b5b-90c853b0d222"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:33.776350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:33.776322 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce8ad726-6306-482d-9b5b-90c853b0d222-kube-api-access-fczjz" pod "ce8ad726-6306-482d-9b5b-90c853b0d222" (UID: "ce8ad726-6306-482d-9b5b-90c853b0d222"). InnerVolumeSpecName "kube-api-access-fczjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:33.776350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:33.776344 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-xks-ca" pod "ce8ad726-6306-482d-9b5b-90c853b0d222" (UID: "ce8ad726-6306-482d-9b5b-90c853b0d222"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:33.876056 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:33.876017 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:33.876056 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:33.876048 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fczjz\" (UniqueName: \"kubernetes.io/projected/ce8ad726-6306-482d-9b5b-90c853b0d222-kube-api-access-fczjz\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:33.876056 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:33.876061 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ce8ad726-6306-482d-9b5b-90c853b0d222-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:34.520033 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:34.520001 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" Oct 30 17:17:34.520033 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:34.520022 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt" event={"ID":"ce8ad726-6306-482d-9b5b-90c853b0d222","Type":"ContainerDied","Data":"42ff829503ea164186b8a9de0343d9deb242ddf5d8c3885e7a7c4ed8fb8ee758"} Oct 30 17:17:34.520227 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:34.520070 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="42ff829503ea164186b8a9de0343d9deb242ddf5d8c3885e7a7c4ed8fb8ee758" Oct 30 17:17:35.358900 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.358839 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" containerID="cri-o://8ad455e79b95b7edde2224c02fee74bf5f39d05de1e48f84497073f824e56a29" gracePeriod=45 Oct 30 17:17:35.372482 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.372421 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" containerID="cri-o://586bf37b7422b3b334c2a4b5c3d2a6374a84404dbd3532256f6bc4345a6c0a73" gracePeriod=45 Oct 30 17:17:35.384103 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.384068 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" containerID="cri-o://1b795b041292e26e1bcf6a4e0642022341e43d6adcd92b58987397c774e59571" gracePeriod=45 Oct 30 17:17:35.528844 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.528334 6400 generic.go:356] "Generic (PLEG): container finished" podID="628efd77-1a83-427b-996a-9495a5a74ed3" containerID="586bf37b7422b3b334c2a4b5c3d2a6374a84404dbd3532256f6bc4345a6c0a73" exitCode=0 Oct 30 17:17:35.528844 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.528399 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" event={"ID":"628efd77-1a83-427b-996a-9495a5a74ed3","Type":"ContainerDied","Data":"586bf37b7422b3b334c2a4b5c3d2a6374a84404dbd3532256f6bc4345a6c0a73"} Oct 30 17:17:35.530148 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.530106 6400 generic.go:356] "Generic (PLEG): container finished" podID="a79cf7f9-6003-44e3-8064-eca14f082220" containerID="8ad455e79b95b7edde2224c02fee74bf5f39d05de1e48f84497073f824e56a29" exitCode=0 Oct 30 17:17:35.530303 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.530171 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" event={"ID":"a79cf7f9-6003-44e3-8064-eca14f082220","Type":"ContainerDied","Data":"8ad455e79b95b7edde2224c02fee74bf5f39d05de1e48f84497073f824e56a29"} Oct 30 17:17:35.531736 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.531710 6400 generic.go:356] "Generic (PLEG): container finished" podID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerID="1b795b041292e26e1bcf6a4e0642022341e43d6adcd92b58987397c774e59571" exitCode=0 Oct 30 17:17:35.531822 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.531812 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" event={"ID":"c890bd5e-2813-4f86-ab91-61f5334d2f2b","Type":"ContainerDied","Data":"1b795b041292e26e1bcf6a4e0642022341e43d6adcd92b58987397c774e59571"} Oct 30 17:17:35.635687 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.635663 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 17:17:35.669890 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.669869 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 17:17:35.673210 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.673186 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 17:17:35.694136 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.694102 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-tmp-dir\") pod \"628efd77-1a83-427b-996a-9495a5a74ed3\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " Oct 30 17:17:35.694302 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.694203 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-dshm\") pod \"628efd77-1a83-427b-996a-9495a5a74ed3\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " Oct 30 17:17:35.694302 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.694282 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-home\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-home\") pod \"628efd77-1a83-427b-996a-9495a5a74ed3\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " Oct 30 17:17:35.694416 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.694323 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/628efd77-1a83-427b-996a-9495a5a74ed3-kube-api-access-kfn6t\" (UniqueName: \"kubernetes.io/projected/628efd77-1a83-427b-996a-9495a5a74ed3-kube-api-access-kfn6t\") pod \"628efd77-1a83-427b-996a-9495a5a74ed3\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " Oct 30 17:17:35.694416 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.694356 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/628efd77-1a83-427b-996a-9495a5a74ed3-tls-certs\" (UniqueName: \"kubernetes.io/secret/628efd77-1a83-427b-996a-9495a5a74ed3-tls-certs\") pod \"628efd77-1a83-427b-996a-9495a5a74ed3\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " Oct 30 17:17:35.694416 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.694399 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-model-cache\") pod \"628efd77-1a83-427b-996a-9495a5a74ed3\" (UID: \"628efd77-1a83-427b-996a-9495a5a74ed3\") " Oct 30 17:17:35.694583 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.694426 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-tmp-dir" pod "628efd77-1a83-427b-996a-9495a5a74ed3" (UID: "628efd77-1a83-427b-996a-9495a5a74ed3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.694852 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.694824 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.695068 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.695043 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-model-cache" pod "628efd77-1a83-427b-996a-9495a5a74ed3" (UID: "628efd77-1a83-427b-996a-9495a5a74ed3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.695326 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.695291 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-home" pod "628efd77-1a83-427b-996a-9495a5a74ed3" (UID: "628efd77-1a83-427b-996a-9495a5a74ed3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.697056 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.697032 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/628efd77-1a83-427b-996a-9495a5a74ed3-tls-certs" pod "628efd77-1a83-427b-996a-9495a5a74ed3" (UID: "628efd77-1a83-427b-996a-9495a5a74ed3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:35.697574 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.697550 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/628efd77-1a83-427b-996a-9495a5a74ed3-kube-api-access-kfn6t" pod "628efd77-1a83-427b-996a-9495a5a74ed3" (UID: "628efd77-1a83-427b-996a-9495a5a74ed3"). InnerVolumeSpecName "kube-api-access-kfn6t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:35.697670 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.697569 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-dshm" pod "628efd77-1a83-427b-996a-9495a5a74ed3" (UID: "628efd77-1a83-427b-996a-9495a5a74ed3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.795454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795422 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a79cf7f9-6003-44e3-8064-eca14f082220-tls-certs\" (UniqueName: \"kubernetes.io/secret/a79cf7f9-6003-44e3-8064-eca14f082220-tls-certs\") pod \"a79cf7f9-6003-44e3-8064-eca14f082220\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " Oct 30 17:17:35.795662 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795474 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-tmp-dir\") pod \"a79cf7f9-6003-44e3-8064-eca14f082220\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " Oct 30 17:17:35.795662 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795501 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c890bd5e-2813-4f86-ab91-61f5334d2f2b-kube-api-access-ltmgp\" (UniqueName: \"kubernetes.io/projected/c890bd5e-2813-4f86-ab91-61f5334d2f2b-kube-api-access-ltmgp\") pod \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " Oct 30 17:17:35.795662 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795530 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-home\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-home\") pod \"a79cf7f9-6003-44e3-8064-eca14f082220\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " Oct 30 17:17:35.795845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795711 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a79cf7f9-6003-44e3-8064-eca14f082220-kube-api-access-jlnbs\" (UniqueName: \"kubernetes.io/projected/a79cf7f9-6003-44e3-8064-eca14f082220-kube-api-access-jlnbs\") pod \"a79cf7f9-6003-44e3-8064-eca14f082220\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " Oct 30 17:17:35.795845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795809 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-dshm\") pod \"a79cf7f9-6003-44e3-8064-eca14f082220\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " Oct 30 17:17:35.795946 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795850 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-home\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-home\") pod \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " Oct 30 17:17:35.795946 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795849 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-tmp-dir" pod "a79cf7f9-6003-44e3-8064-eca14f082220" (UID: "a79cf7f9-6003-44e3-8064-eca14f082220"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.795946 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795860 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-home" pod "a79cf7f9-6003-44e3-8064-eca14f082220" (UID: "a79cf7f9-6003-44e3-8064-eca14f082220"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.795946 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795916 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-dshm\") pod \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " Oct 30 17:17:35.796149 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.795967 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-model-cache\") pod \"a79cf7f9-6003-44e3-8064-eca14f082220\" (UID: \"a79cf7f9-6003-44e3-8064-eca14f082220\") " Oct 30 17:17:35.796149 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796066 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tls-certs\" (UniqueName: \"kubernetes.io/secret/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tls-certs\") pod \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " Oct 30 17:17:35.796149 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796071 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-home" pod "c890bd5e-2813-4f86-ab91-61f5334d2f2b" (UID: "c890bd5e-2813-4f86-ab91-61f5334d2f2b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.796149 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796102 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-model-cache\") pod \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " Oct 30 17:17:35.796357 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796160 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tmp-dir\") pod \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\" (UID: \"c890bd5e-2813-4f86-ab91-61f5334d2f2b\") " Oct 30 17:17:35.797075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796523 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.797075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796548 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfn6t\" (UniqueName: \"kubernetes.io/projected/628efd77-1a83-427b-996a-9495a5a74ed3-kube-api-access-kfn6t\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.797075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796563 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/628efd77-1a83-427b-996a-9495a5a74ed3-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.797075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796577 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.797075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796637 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.797075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796732 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.797075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796788 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/628efd77-1a83-427b-996a-9495a5a74ed3-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.797075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.796800 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.797075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.797012 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-model-cache" pod "c890bd5e-2813-4f86-ab91-61f5334d2f2b" (UID: "c890bd5e-2813-4f86-ab91-61f5334d2f2b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.797075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.797022 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tmp-dir" pod "c890bd5e-2813-4f86-ab91-61f5334d2f2b" (UID: "c890bd5e-2813-4f86-ab91-61f5334d2f2b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.797759 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.797223 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-model-cache" pod "a79cf7f9-6003-44e3-8064-eca14f082220" (UID: "a79cf7f9-6003-44e3-8064-eca14f082220"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.798740 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.798718 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a79cf7f9-6003-44e3-8064-eca14f082220-tls-certs" pod "a79cf7f9-6003-44e3-8064-eca14f082220" (UID: "a79cf7f9-6003-44e3-8064-eca14f082220"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:35.799030 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.798752 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-dshm" pod "a79cf7f9-6003-44e3-8064-eca14f082220" (UID: "a79cf7f9-6003-44e3-8064-eca14f082220"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.799030 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.798794 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a79cf7f9-6003-44e3-8064-eca14f082220-kube-api-access-jlnbs" pod "a79cf7f9-6003-44e3-8064-eca14f082220" (UID: "a79cf7f9-6003-44e3-8064-eca14f082220"). InnerVolumeSpecName "kube-api-access-jlnbs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:35.799030 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.798830 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c890bd5e-2813-4f86-ab91-61f5334d2f2b-kube-api-access-ltmgp" pod "c890bd5e-2813-4f86-ab91-61f5334d2f2b" (UID: "c890bd5e-2813-4f86-ab91-61f5334d2f2b"). InnerVolumeSpecName "kube-api-access-ltmgp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:35.799299 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.799279 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tls-certs" pod "c890bd5e-2813-4f86-ab91-61f5334d2f2b" (UID: "c890bd5e-2813-4f86-ab91-61f5334d2f2b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:35.800259 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.800232 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-dshm" pod "c890bd5e-2813-4f86-ab91-61f5334d2f2b" (UID: "c890bd5e-2813-4f86-ab91-61f5334d2f2b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:35.897678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.897559 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.897678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.897590 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a79cf7f9-6003-44e3-8064-eca14f082220-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.897678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.897633 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltmgp\" (UniqueName: \"kubernetes.io/projected/c890bd5e-2813-4f86-ab91-61f5334d2f2b-kube-api-access-ltmgp\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.897678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.897647 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jlnbs\" (UniqueName: \"kubernetes.io/projected/a79cf7f9-6003-44e3-8064-eca14f082220-kube-api-access-jlnbs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.897678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.897660 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.897678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.897673 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.898007 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.897684 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a79cf7f9-6003-44e3-8064-eca14f082220-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.898007 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.897696 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c890bd5e-2813-4f86-ab91-61f5334d2f2b-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:35.898007 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:35.897718 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c890bd5e-2813-4f86-ab91-61f5334d2f2b-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:36.539129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.539096 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" Oct 30 17:17:36.539708 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.539101 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" event={"ID":"a79cf7f9-6003-44e3-8064-eca14f082220","Type":"ContainerDied","Data":"9ea7cac6f21c1ecdb975791105e196f06606cf4c5dbf28ee6c9807ecc6fad917"} Oct 30 17:17:36.539708 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.539238 6400 scope.go:122] "RemoveContainer" containerID="8ad455e79b95b7edde2224c02fee74bf5f39d05de1e48f84497073f824e56a29" Oct 30 17:17:36.540722 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.540654 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" Oct 30 17:17:36.540722 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.540707 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" event={"ID":"c890bd5e-2813-4f86-ab91-61f5334d2f2b","Type":"ContainerDied","Data":"d4931813c312884f5fc32010416004ac94efe9e5f990a1522d6104a0691650e5"} Oct 30 17:17:36.542398 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.542376 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" Oct 30 17:17:36.542398 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.542393 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" event={"ID":"628efd77-1a83-427b-996a-9495a5a74ed3","Type":"ContainerDied","Data":"0fede5c281b0f1f2b96bb282d158d506ff2499768d297fc43e8af01731b4815c"} Oct 30 17:17:36.550543 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.550526 6400 scope.go:122] "RemoveContainer" containerID="1b795b041292e26e1bcf6a4e0642022341e43d6adcd92b58987397c774e59571" Oct 30 17:17:36.560132 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.560112 6400 scope.go:122] "RemoveContainer" containerID="586bf37b7422b3b334c2a4b5c3d2a6374a84404dbd3532256f6bc4345a6c0a73" Oct 30 17:17:36.588148 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.588113 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" probeResult="failure" output="Get \"https://10.134.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:17:36.593846 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.593819 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8"] Oct 30 17:17:36.605886 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.605830 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 17:17:36.606846 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.606825 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8"] Oct 30 17:17:36.632566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.632491 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z"] Oct 30 17:17:36.634586 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.634569 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/premium-w2-750dcb-kserve-667d65c7b7-zd98z"] Oct 30 17:17:36.652308 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.652276 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-750dcb/simulator-w2-750dcb-kserve-8649fb579d-m4wn8" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded" Oct 30 17:17:36.670499 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.670473 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d"] Oct 30 17:17:36.675567 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.675541 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-750dcb/distinct-2-w2-750dcb-kserve-6fc979594f-5vw5d"] Oct 30 17:17:36.965440 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.965350 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" path="/var/lib/kubelet/pods/628efd77-1a83-427b-996a-9495a5a74ed3/volumes" Oct 30 17:17:36.965863 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.965843 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" path="/var/lib/kubelet/pods/a79cf7f9-6003-44e3-8064-eca14f082220/volumes" Oct 30 17:17:36.966262 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:36.966244 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" path="/var/lib/kubelet/pods/c890bd5e-2813-4f86-ab91-61f5334d2f2b/volumes" Oct 30 17:17:38.984894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:38.984856 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn"] Oct 30 17:17:38.985364 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:38.985075 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerName="istio-proxy" containerID="cri-o://8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db" gracePeriod=30 Oct 30 17:17:39.694376 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694348 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt"] Oct 30 17:17:39.694743 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694730 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" Oct 30 17:17:39.694793 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694745 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" Oct 30 17:17:39.694793 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694777 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" Oct 30 17:17:39.694793 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694782 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" Oct 30 17:17:39.694793 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694792 6400 cpu_manager.go:414] "containerMap: removing container" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" Oct 30 17:17:39.694926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694798 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" Oct 30 17:17:39.694926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694807 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a4653719-1cc9-451a-8634-fa8924d6ac41" containerName="maas-api" Oct 30 17:17:39.694926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694812 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4653719-1cc9-451a-8634-fa8924d6ac41" containerName="maas-api" Oct 30 17:17:39.694926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694824 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ce8ad726-6306-482d-9b5b-90c853b0d222" containerName="revoke-keys" Oct 30 17:17:39.694926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694829 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ce8ad726-6306-482d-9b5b-90c853b0d222" containerName="revoke-keys" Oct 30 17:17:39.694926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694877 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="628efd77-1a83-427b-996a-9495a5a74ed3" containerName="main" Oct 30 17:17:39.694926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694888 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4653719-1cc9-451a-8634-fa8924d6ac41" containerName="maas-api" Oct 30 17:17:39.694926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694896 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ce8ad726-6306-482d-9b5b-90c853b0d222" containerName="revoke-keys" Oct 30 17:17:39.694926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694902 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a79cf7f9-6003-44e3-8064-eca14f082220" containerName="main" Oct 30 17:17:39.694926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.694912 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c890bd5e-2813-4f86-ab91-61f5334d2f2b" containerName="main" Oct 30 17:17:39.699740 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.699723 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:39.720257 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.720219 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt"] Oct 30 17:17:39.727730 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.727692 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:17:39.731178 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.731161 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:17:39.835937 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.835903 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:39.836202 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.835943 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:39.836202 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.836084 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2k9b\" (UniqueName: \"kubernetes.io/projected/e4be1dda-22e4-41d5-9e00-835a86e0272d-kube-api-access-b2k9b\") pod \"maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:39.937521 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.937486 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2k9b\" (UniqueName: \"kubernetes.io/projected/e4be1dda-22e4-41d5-9e00-835a86e0272d-kube-api-access-b2k9b\") pod \"maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:39.937727 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.937625 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:39.937727 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.937663 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:39.938348 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.938316 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:39.940185 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.940167 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:39.962703 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:39.962639 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2k9b\" (UniqueName: \"kubernetes.io/projected/e4be1dda-22e4-41d5-9e00-835a86e0272d-kube-api-access-b2k9b\") pod \"maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:40.012498 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:40.012467 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:40.311553 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:40.311523 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt"] Oct 30 17:17:40.313196 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:17:40.313169 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4be1dda_22e4_41d5_9e00_835a86e0272d.slice/crio-90a774a6658cefa63aeb6d62520ca2add1f09df5bea6b452abf0378e15406e6d WatchSource:0}: Error finding container 90a774a6658cefa63aeb6d62520ca2add1f09df5bea6b452abf0378e15406e6d: Status 404 returned error can't find the container with id 90a774a6658cefa63aeb6d62520ca2add1f09df5bea6b452abf0378e15406e6d Oct 30 17:17:40.567509 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:40.567424 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" event={"ID":"e4be1dda-22e4-41d5-9e00-835a86e0272d","Type":"ContainerStarted","Data":"90a774a6658cefa63aeb6d62520ca2add1f09df5bea6b452abf0378e15406e6d"} Oct 30 17:17:41.574157 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:41.574124 6400 generic.go:356] "Generic (PLEG): container finished" podID="e4be1dda-22e4-41d5-9e00-835a86e0272d" containerID="bdc4ca6d98038ccf2c172e6444d6031e359a921059e78ced6f64c398dae0e020" exitCode=0 Oct 30 17:17:41.574568 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:41.574206 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" event={"ID":"e4be1dda-22e4-41d5-9e00-835a86e0272d","Type":"ContainerDied","Data":"bdc4ca6d98038ccf2c172e6444d6031e359a921059e78ced6f64c398dae0e020"} Oct 30 17:17:42.717833 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:42.717805 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:42.864338 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:42.864252 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-xks-ca\") pod \"e4be1dda-22e4-41d5-9e00-835a86e0272d\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " Oct 30 17:17:42.864338 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:42.864321 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-openshift-ca\") pod \"e4be1dda-22e4-41d5-9e00-835a86e0272d\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " Oct 30 17:17:42.864545 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:42.864441 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4be1dda-22e4-41d5-9e00-835a86e0272d-kube-api-access-b2k9b\" (UniqueName: \"kubernetes.io/projected/e4be1dda-22e4-41d5-9e00-835a86e0272d-kube-api-access-b2k9b\") pod \"e4be1dda-22e4-41d5-9e00-835a86e0272d\" (UID: \"e4be1dda-22e4-41d5-9e00-835a86e0272d\") " Oct 30 17:17:42.864786 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:42.864762 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-openshift-ca" pod "e4be1dda-22e4-41d5-9e00-835a86e0272d" (UID: "e4be1dda-22e4-41d5-9e00-835a86e0272d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:42.866668 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:42.866638 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-xks-ca" pod "e4be1dda-22e4-41d5-9e00-835a86e0272d" (UID: "e4be1dda-22e4-41d5-9e00-835a86e0272d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:42.866769 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:42.866705 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4be1dda-22e4-41d5-9e00-835a86e0272d-kube-api-access-b2k9b" pod "e4be1dda-22e4-41d5-9e00-835a86e0272d" (UID: "e4be1dda-22e4-41d5-9e00-835a86e0272d"). InnerVolumeSpecName "kube-api-access-b2k9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:42.965799 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:42.965767 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2k9b\" (UniqueName: \"kubernetes.io/projected/e4be1dda-22e4-41d5-9e00-835a86e0272d-kube-api-access-b2k9b\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:42.965799 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:42.965796 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:42.965799 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:42.965805 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e4be1dda-22e4-41d5-9e00-835a86e0272d-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:43.585388 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:43.585348 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" event={"ID":"e4be1dda-22e4-41d5-9e00-835a86e0272d","Type":"ContainerDied","Data":"90a774a6658cefa63aeb6d62520ca2add1f09df5bea6b452abf0378e15406e6d"} Oct 30 17:17:43.585388 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:43.585371 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt" Oct 30 17:17:43.585671 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:43.585396 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="90a774a6658cefa63aeb6d62520ca2add1f09df5bea6b452abf0378e15406e6d" Oct 30 17:17:44.247385 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.247362 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:17:44.378920 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.378836 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-envoy\") pod \"61d6bcdd-9469-4086-af49-e19ff94217e9\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " Oct 30 17:17:44.378920 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.378884 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-credential-socket\") pod \"61d6bcdd-9469-4086-af49-e19ff94217e9\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " Oct 30 17:17:44.379163 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.378922 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-data\") pod \"61d6bcdd-9469-4086-af49-e19ff94217e9\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " Oct 30 17:17:44.379163 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.378982 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-podinfo\") pod \"61d6bcdd-9469-4086-af49-e19ff94217e9\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " Oct 30 17:17:44.379163 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379024 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-token\" (UniqueName: \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-token\") pod \"61d6bcdd-9469-4086-af49-e19ff94217e9\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " Oct 30 17:17:44.379163 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379063 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d6bcdd-9469-4086-af49-e19ff94217e9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d6bcdd-9469-4086-af49-e19ff94217e9-istiod-ca-cert\") pod \"61d6bcdd-9469-4086-af49-e19ff94217e9\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " Oct 30 17:17:44.379163 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379089 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-socket\") pod \"61d6bcdd-9469-4086-af49-e19ff94217e9\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " Oct 30 17:17:44.379163 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379114 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-certs\") pod \"61d6bcdd-9469-4086-af49-e19ff94217e9\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " Oct 30 17:17:44.379472 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379209 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-kube-api-access-ctzvm\" (UniqueName: \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-kube-api-access-ctzvm\") pod \"61d6bcdd-9469-4086-af49-e19ff94217e9\" (UID: \"61d6bcdd-9469-4086-af49-e19ff94217e9\") " Oct 30 17:17:44.379472 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379282 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-credential-socket" pod "61d6bcdd-9469-4086-af49-e19ff94217e9" (UID: "61d6bcdd-9469-4086-af49-e19ff94217e9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:44.379472 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379293 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-data" pod "61d6bcdd-9469-4086-af49-e19ff94217e9" (UID: "61d6bcdd-9469-4086-af49-e19ff94217e9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:44.379472 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379435 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-socket" pod "61d6bcdd-9469-4086-af49-e19ff94217e9" (UID: "61d6bcdd-9469-4086-af49-e19ff94217e9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:44.379692 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379521 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-certs" pod "61d6bcdd-9469-4086-af49-e19ff94217e9" (UID: "61d6bcdd-9469-4086-af49-e19ff94217e9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:44.379692 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379554 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d6bcdd-9469-4086-af49-e19ff94217e9-istiod-ca-cert" pod "61d6bcdd-9469-4086-af49-e19ff94217e9" (UID: "61d6bcdd-9469-4086-af49-e19ff94217e9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:44.379692 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379663 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.379692 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379676 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.379692 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379685 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.379692 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.379693 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.381655 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.381589 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-token" pod "61d6bcdd-9469-4086-af49-e19ff94217e9" (UID: "61d6bcdd-9469-4086-af49-e19ff94217e9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:44.381788 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.381707 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-kube-api-access-ctzvm" pod "61d6bcdd-9469-4086-af49-e19ff94217e9" (UID: "61d6bcdd-9469-4086-af49-e19ff94217e9"). InnerVolumeSpecName "kube-api-access-ctzvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:44.381836 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.381803 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-envoy" pod "61d6bcdd-9469-4086-af49-e19ff94217e9" (UID: "61d6bcdd-9469-4086-af49-e19ff94217e9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:44.382028 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.382013 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-podinfo" pod "61d6bcdd-9469-4086-af49-e19ff94217e9" (UID: "61d6bcdd-9469-4086-af49-e19ff94217e9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:17:44.480199 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.480168 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.480199 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.480194 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.480199 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.480203 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.480453 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.480211 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/61d6bcdd-9469-4086-af49-e19ff94217e9-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.480453 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.480220 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ctzvm\" (UniqueName: \"kubernetes.io/projected/61d6bcdd-9469-4086-af49-e19ff94217e9-kube-api-access-ctzvm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:44.550663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.550626 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq"] Oct 30 17:17:44.558310 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.558283 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb"] Oct 30 17:17:44.582215 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.582184 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj"] Oct 30 17:17:44.594951 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.594914 6400 generic.go:356] "Generic (PLEG): container finished" podID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerID="8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db" exitCode=0 Oct 30 17:17:44.595148 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.594984 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" Oct 30 17:17:44.595148 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.595015 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" event={"ID":"61d6bcdd-9469-4086-af49-e19ff94217e9","Type":"ContainerDied","Data":"8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db"} Oct 30 17:17:44.595148 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.595075 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn" event={"ID":"61d6bcdd-9469-4086-af49-e19ff94217e9","Type":"ContainerDied","Data":"401b5b4e627b7fbe4fdb266571f197cce223f603cb5ec88abe5dc0d3c5657195"} Oct 30 17:17:44.595148 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.595110 6400 scope.go:122] "RemoveContainer" containerID="8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db" Oct 30 17:17:44.611262 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.611231 6400 scope.go:122] "RemoveContainer" containerID="8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db" Oct 30 17:17:44.611537 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:44.611521 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db\": container with ID starting with 8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db not found: ID does not exist" containerID="8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db" Oct 30 17:17:44.611591 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.611548 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db"} err="failed to get container status \"8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db\": rpc error: code = NotFound desc = could not find container \"8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db\": container with ID starting with 8611286642cde5e0bc1d2b7b7225a1aad32a5556b53c5618244c8e2fa75794db not found: ID does not exist" Oct 30 17:17:44.742237 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.742207 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn"] Oct 30 17:17:44.746026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.745994 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-9e533f-openshift-default-557966b85f-xfnfn"] Oct 30 17:17:44.966902 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:44.966867 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" path="/var/lib/kubelet/pods/61d6bcdd-9469-4086-af49-e19ff94217e9/volumes" Oct 30 17:17:51.201171 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.201136 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms"] Oct 30 17:17:51.201560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.201491 6400 cpu_manager.go:414] "containerMap: removing container" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerName="istio-proxy" Oct 30 17:17:51.201560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.201501 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerName="istio-proxy" Oct 30 17:17:51.201560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.201523 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e4be1dda-22e4-41d5-9e00-835a86e0272d" containerName="revoke-keys" Oct 30 17:17:51.201560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.201530 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4be1dda-22e4-41d5-9e00-835a86e0272d" containerName="revoke-keys" Oct 30 17:17:51.201753 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.201596 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4be1dda-22e4-41d5-9e00-835a86e0272d" containerName="revoke-keys" Oct 30 17:17:51.201753 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.201624 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d6bcdd-9469-4086-af49-e19ff94217e9" containerName="istio-proxy" Oct 30 17:17:51.206028 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.206008 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.238174 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.238142 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:17:51.238374 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.238276 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:17:51.240116 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.240093 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms"] Oct 30 17:17:51.350894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.350860 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cd8bb6e-fjlms\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.351047 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.350912 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mmxh6\" (UniqueName: \"kubernetes.io/projected/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-kube-api-access-mmxh6\") pod \"maas-api-revoke-sub-3cd8bb6e-fjlms\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.351047 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.351025 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cd8bb6e-fjlms\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.451677 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.451555 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cd8bb6e-fjlms\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.451677 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.451633 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mmxh6\" (UniqueName: \"kubernetes.io/projected/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-kube-api-access-mmxh6\") pod \"maas-api-revoke-sub-3cd8bb6e-fjlms\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.451677 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.451667 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cd8bb6e-fjlms\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.452239 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.452212 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3cd8bb6e-fjlms\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.454173 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.454146 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3cd8bb6e-fjlms\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.476310 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.476281 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mmxh6\" (UniqueName: \"kubernetes.io/projected/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-kube-api-access-mmxh6\") pod \"maas-api-revoke-sub-3cd8bb6e-fjlms\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.521451 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.521420 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:51.820582 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:51.820554 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms"] Oct 30 17:17:51.821396 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:17:51.821371 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdc0a9eb_8c2c_40e0_a3d7_e4cdda97a256.slice/crio-9153e2e6989af6dbf2c942f96b5eeabb06f94d44057b4da281524103406368ba WatchSource:0}: Error finding container 9153e2e6989af6dbf2c942f96b5eeabb06f94d44057b4da281524103406368ba: Status 404 returned error can't find the container with id 9153e2e6989af6dbf2c942f96b5eeabb06f94d44057b4da281524103406368ba Oct 30 17:17:52.641413 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:52.641371 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" event={"ID":"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256","Type":"ContainerStarted","Data":"0054b8e5d56fc608f272848fd48ed66dd58de9273a79357f99becc7167461248"} Oct 30 17:17:52.641940 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:52.641424 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" event={"ID":"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256","Type":"ContainerStarted","Data":"9153e2e6989af6dbf2c942f96b5eeabb06f94d44057b4da281524103406368ba"} Oct 30 17:17:52.684016 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:52.683965 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" podStartSLOduration=1.683950853 podStartE2EDuration="1.683950853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:17:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:17:52.683108671 +0000 UTC m=+3381.833389659" watchObservedRunningTime="2026-10-30 17:17:52.683950853 +0000 UTC m=+3381.834231878" Oct 30 17:17:53.647990 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:53.647953 6400 generic.go:356] "Generic (PLEG): container finished" podID="fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256" containerID="0054b8e5d56fc608f272848fd48ed66dd58de9273a79357f99becc7167461248" exitCode=0 Oct 30 17:17:53.648461 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:53.648041 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" event={"ID":"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256","Type":"ContainerDied","Data":"0054b8e5d56fc608f272848fd48ed66dd58de9273a79357f99becc7167461248"} Oct 30 17:17:54.596825 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:17:54.596783 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:17:54.796306 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:54.796280 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:54.986033 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:54.986007 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-openshift-ca\") pod \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " Oct 30 17:17:54.986234 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:54.986056 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-xks-ca\") pod \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " Oct 30 17:17:54.986234 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:54.986093 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-kube-api-access-mmxh6\" (UniqueName: \"kubernetes.io/projected/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-kube-api-access-mmxh6\") pod \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\" (UID: \"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256\") " Oct 30 17:17:54.986371 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:54.986350 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-openshift-ca" pod "fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256" (UID: "fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:17:54.986509 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:54.986491 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:54.988395 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:54.988370 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-xks-ca" pod "fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256" (UID: "fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:54.988507 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:54.988481 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-kube-api-access-mmxh6" pod "fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256" (UID: "fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256"). InnerVolumeSpecName "kube-api-access-mmxh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:55.087890 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:55.087852 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:55.087890 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:55.087884 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mmxh6\" (UniqueName: \"kubernetes.io/projected/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256-kube-api-access-mmxh6\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:55.660591 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:55.660560 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" Oct 30 17:17:55.660796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:55.660560 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms" event={"ID":"fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256","Type":"ContainerDied","Data":"9153e2e6989af6dbf2c942f96b5eeabb06f94d44057b4da281524103406368ba"} Oct 30 17:17:55.660796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:55.660694 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9153e2e6989af6dbf2c942f96b5eeabb06f94d44057b4da281524103406368ba" Oct 30 17:17:59.572880 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.572810 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" containerID="cri-o://b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f" gracePeriod=45 Oct 30 17:17:59.579258 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.579227 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" containerID="cri-o://d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337" gracePeriod=45 Oct 30 17:17:59.605041 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.604973 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" containerID="cri-o://2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843" gracePeriod=45 Oct 30 17:17:59.618177 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.618145 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" probeResult="failure" output="Get \"https://10.134.0.53:8000/ready\": dial tcp 10.134.0.53:8000: connect: connection refused" Oct 30 17:17:59.757560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.757515 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" probeResult="failure" output="Get \"https://10.134.0.80:8000/ready\": dial tcp 10.134.0.80:8000: connect: connection refused" Oct 30 17:17:59.877163 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.877120 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 17:17:59.879042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.879019 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 17:17:59.893755 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.893731 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 17:17:59.921060 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.921026 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-dshm\") pod \"32cf4325-2f94-455f-82b5-f33ad48c3884\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " Oct 30 17:17:59.921060 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.921087 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-dshm\") pod \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " Oct 30 17:17:59.921369 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.921127 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-model-cache\") pod \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " Oct 30 17:17:59.921369 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.921173 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-model-cache\") pod \"32cf4325-2f94-455f-82b5-f33ad48c3884\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " Oct 30 17:17:59.921369 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.921207 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-tmp-dir\") pod \"32cf4325-2f94-455f-82b5-f33ad48c3884\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " Oct 30 17:17:59.921369 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.921238 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-home\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-home\") pod \"6927cc23-033e-4bad-9a12-573afcf72cf0\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " Oct 30 17:17:59.922120 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922093 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6927cc23-033e-4bad-9a12-573afcf72cf0-tls-certs\" (UniqueName: \"kubernetes.io/secret/6927cc23-033e-4bad-9a12-573afcf72cf0-tls-certs\") pod \"6927cc23-033e-4bad-9a12-573afcf72cf0\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " Oct 30 17:17:59.922277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922159 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-dshm\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-dshm\") pod \"6927cc23-033e-4bad-9a12-573afcf72cf0\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " Oct 30 17:17:59.922277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922194 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-tmp-dir\") pod \"6927cc23-033e-4bad-9a12-573afcf72cf0\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " Oct 30 17:17:59.922277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922236 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ae453e0-0c2c-4080-bef3-c2119549a4ef-kube-api-access-fmm85\" (UniqueName: \"kubernetes.io/projected/6ae453e0-0c2c-4080-bef3-c2119549a4ef-kube-api-access-fmm85\") pod \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " Oct 30 17:17:59.922277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922261 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-home\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-home\") pod \"32cf4325-2f94-455f-82b5-f33ad48c3884\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " Oct 30 17:17:59.922566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922305 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32cf4325-2f94-455f-82b5-f33ad48c3884-kube-api-access-chmfz\" (UniqueName: \"kubernetes.io/projected/32cf4325-2f94-455f-82b5-f33ad48c3884-kube-api-access-chmfz\") pod \"32cf4325-2f94-455f-82b5-f33ad48c3884\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " Oct 30 17:17:59.922566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922342 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-model-cache\") pod \"6927cc23-033e-4bad-9a12-573afcf72cf0\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " Oct 30 17:17:59.922566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922373 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32cf4325-2f94-455f-82b5-f33ad48c3884-tls-certs\" (UniqueName: \"kubernetes.io/secret/32cf4325-2f94-455f-82b5-f33ad48c3884-tls-certs\") pod \"32cf4325-2f94-455f-82b5-f33ad48c3884\" (UID: \"32cf4325-2f94-455f-82b5-f33ad48c3884\") " Oct 30 17:17:59.922566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922404 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-home\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-home\") pod \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " Oct 30 17:17:59.922566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922450 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tls-certs\") pod \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " Oct 30 17:17:59.922566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922480 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tmp-dir\") pod \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\" (UID: \"6ae453e0-0c2c-4080-bef3-c2119549a4ef\") " Oct 30 17:17:59.922566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922512 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6927cc23-033e-4bad-9a12-573afcf72cf0-kube-api-access-wzt6w\" (UniqueName: \"kubernetes.io/projected/6927cc23-033e-4bad-9a12-573afcf72cf0-kube-api-access-wzt6w\") pod \"6927cc23-033e-4bad-9a12-573afcf72cf0\" (UID: \"6927cc23-033e-4bad-9a12-573afcf72cf0\") " Oct 30 17:17:59.922987 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.921714 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-model-cache" pod "32cf4325-2f94-455f-82b5-f33ad48c3884" (UID: "32cf4325-2f94-455f-82b5-f33ad48c3884"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.922987 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.921897 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-home" pod "6927cc23-033e-4bad-9a12-573afcf72cf0" (UID: "6927cc23-033e-4bad-9a12-573afcf72cf0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.922987 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.921956 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-model-cache" pod "6ae453e0-0c2c-4080-bef3-c2119549a4ef" (UID: "6ae453e0-0c2c-4080-bef3-c2119549a4ef"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.922987 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922023 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-tmp-dir" pod "32cf4325-2f94-455f-82b5-f33ad48c3884" (UID: "32cf4325-2f94-455f-82b5-f33ad48c3884"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.922987 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922699 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-tmp-dir" pod "6927cc23-033e-4bad-9a12-573afcf72cf0" (UID: "6927cc23-033e-4bad-9a12-573afcf72cf0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.922987 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.922919 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-model-cache" pod "6927cc23-033e-4bad-9a12-573afcf72cf0" (UID: "6927cc23-033e-4bad-9a12-573afcf72cf0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.924803 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.924483 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-home" pod "32cf4325-2f94-455f-82b5-f33ad48c3884" (UID: "32cf4325-2f94-455f-82b5-f33ad48c3884"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.924937 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.924821 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-home" pod "6ae453e0-0c2c-4080-bef3-c2119549a4ef" (UID: "6ae453e0-0c2c-4080-bef3-c2119549a4ef"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925127 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tmp-dir" pod "6ae453e0-0c2c-4080-bef3-c2119549a4ef" (UID: "6ae453e0-0c2c-4080-bef3-c2119549a4ef"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925383 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925400 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925412 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925423 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925434 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925447 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925460 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925475 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925487 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:17:59.928948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.925951 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cf4325-2f94-455f-82b5-f33ad48c3884-tls-certs" pod "32cf4325-2f94-455f-82b5-f33ad48c3884" (UID: "32cf4325-2f94-455f-82b5-f33ad48c3884"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:59.930513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.930470 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6927cc23-033e-4bad-9a12-573afcf72cf0-tls-certs" pod "6927cc23-033e-4bad-9a12-573afcf72cf0" (UID: "6927cc23-033e-4bad-9a12-573afcf72cf0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:59.930663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.930509 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-dshm" pod "6ae453e0-0c2c-4080-bef3-c2119549a4ef" (UID: "6ae453e0-0c2c-4080-bef3-c2119549a4ef"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.930663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.930513 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ae453e0-0c2c-4080-bef3-c2119549a4ef-kube-api-access-fmm85" pod "6ae453e0-0c2c-4080-bef3-c2119549a4ef" (UID: "6ae453e0-0c2c-4080-bef3-c2119549a4ef"). InnerVolumeSpecName "kube-api-access-fmm85". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:59.931494 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.931448 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-dshm" pod "32cf4325-2f94-455f-82b5-f33ad48c3884" (UID: "32cf4325-2f94-455f-82b5-f33ad48c3884"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.931694 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.931642 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cf4325-2f94-455f-82b5-f33ad48c3884-kube-api-access-chmfz" pod "32cf4325-2f94-455f-82b5-f33ad48c3884" (UID: "32cf4325-2f94-455f-82b5-f33ad48c3884"). InnerVolumeSpecName "kube-api-access-chmfz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:17:59.932502 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.932476 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-dshm" pod "6927cc23-033e-4bad-9a12-573afcf72cf0" (UID: "6927cc23-033e-4bad-9a12-573afcf72cf0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:17:59.932962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.932933 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tls-certs" pod "6ae453e0-0c2c-4080-bef3-c2119549a4ef" (UID: "6ae453e0-0c2c-4080-bef3-c2119549a4ef"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:17:59.934303 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:17:59.934282 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6927cc23-033e-4bad-9a12-573afcf72cf0-kube-api-access-wzt6w" pod "6927cc23-033e-4bad-9a12-573afcf72cf0" (UID: "6927cc23-033e-4bad-9a12-573afcf72cf0"). InnerVolumeSpecName "kube-api-access-wzt6w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:18:00.027078 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.027048 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6927cc23-033e-4bad-9a12-573afcf72cf0-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.027078 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.027077 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmm85\" (UniqueName: \"kubernetes.io/projected/6ae453e0-0c2c-4080-bef3-c2119549a4ef-kube-api-access-fmm85\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.027340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.027088 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chmfz\" (UniqueName: \"kubernetes.io/projected/32cf4325-2f94-455f-82b5-f33ad48c3884-kube-api-access-chmfz\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.027340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.027098 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/32cf4325-2f94-455f-82b5-f33ad48c3884-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.027340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.027107 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ae453e0-0c2c-4080-bef3-c2119549a4ef-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.027340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.027115 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzt6w\" (UniqueName: \"kubernetes.io/projected/6927cc23-033e-4bad-9a12-573afcf72cf0-kube-api-access-wzt6w\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.027340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.027122 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/32cf4325-2f94-455f-82b5-f33ad48c3884-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.027340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.027130 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ae453e0-0c2c-4080-bef3-c2119549a4ef-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.027340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.027137 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6927cc23-033e-4bad-9a12-573afcf72cf0-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:00.690776 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.690739 6400 generic.go:356] "Generic (PLEG): container finished" podID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerID="d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337" exitCode=0 Oct 30 17:18:00.691245 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.690782 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" Oct 30 17:18:00.691245 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.690826 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" event={"ID":"6927cc23-033e-4bad-9a12-573afcf72cf0","Type":"ContainerDied","Data":"d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337"} Oct 30 17:18:00.691245 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.690879 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb" event={"ID":"6927cc23-033e-4bad-9a12-573afcf72cf0","Type":"ContainerDied","Data":"4828f85b06bb5166d400ac227bf5e915ef72976b2bef897ea3eac67081943719"} Oct 30 17:18:00.691245 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.690895 6400 scope.go:122] "RemoveContainer" containerID="d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337" Oct 30 17:18:00.692343 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.692325 6400 generic.go:356] "Generic (PLEG): container finished" podID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerID="b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f" exitCode=0 Oct 30 17:18:00.692423 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.692362 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" Oct 30 17:18:00.692470 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.692411 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" event={"ID":"6ae453e0-0c2c-4080-bef3-c2119549a4ef","Type":"ContainerDied","Data":"b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f"} Oct 30 17:18:00.692714 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.692613 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq" event={"ID":"6ae453e0-0c2c-4080-bef3-c2119549a4ef","Type":"ContainerDied","Data":"55d4f3f06591b412d9379c61b6c49de1ae3ec7d8d48bdaf7137c1f951929ccd1"} Oct 30 17:18:00.694286 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.694249 6400 generic.go:356] "Generic (PLEG): container finished" podID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerID="2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843" exitCode=0 Oct 30 17:18:00.694397 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.694291 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" Oct 30 17:18:00.694837 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.694295 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" event={"ID":"32cf4325-2f94-455f-82b5-f33ad48c3884","Type":"ContainerDied","Data":"2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843"} Oct 30 17:18:00.694921 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.694859 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj" event={"ID":"32cf4325-2f94-455f-82b5-f33ad48c3884","Type":"ContainerDied","Data":"fdc50db3f28540e9235118fe9e9dfdc9dbeadaa33d9f0ca989bd6e49cd15f994"} Oct 30 17:18:00.702982 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.702958 6400 scope.go:122] "RemoveContainer" containerID="d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337" Oct 30 17:18:00.703280 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:18:00.703260 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337\": container with ID starting with d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337 not found: ID does not exist" containerID="d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337" Oct 30 17:18:00.703368 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.703287 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337"} err="failed to get container status \"d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337\": rpc error: code = NotFound desc = could not find container \"d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337\": container with ID starting with d42a90e6f91f3ef146e9b3ae7ec74c44eeb97c22daee41d2b3a809516828a337 not found: ID does not exist" Oct 30 17:18:00.703368 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.703306 6400 scope.go:122] "RemoveContainer" containerID="b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f" Oct 30 17:18:00.712918 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.712897 6400 scope.go:122] "RemoveContainer" containerID="b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f" Oct 30 17:18:00.713181 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:18:00.713163 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f\": container with ID starting with b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f not found: ID does not exist" containerID="b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f" Oct 30 17:18:00.713253 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.713190 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f"} err="failed to get container status \"b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f\": rpc error: code = NotFound desc = could not find container \"b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f\": container with ID starting with b097d1392473e4e1c446e38a1f22a239ff0c24cb71fff9e6a3b591fbd58fcd8f not found: ID does not exist" Oct 30 17:18:00.713253 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.713209 6400 scope.go:122] "RemoveContainer" containerID="2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843" Oct 30 17:18:00.721305 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.721288 6400 scope.go:122] "RemoveContainer" containerID="2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843" Oct 30 17:18:00.721539 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:18:00.721522 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843\": container with ID starting with 2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843 not found: ID does not exist" containerID="2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843" Oct 30 17:18:00.721586 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.721546 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843"} err="failed to get container status \"2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843\": rpc error: code = NotFound desc = could not find container \"2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843\": container with ID starting with 2f043745eda3498616bdbed9a39c81995f7440c549eec6271efbfaa77c147843 not found: ID does not exist" Oct 30 17:18:00.744731 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.744707 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb"] Oct 30 17:18:00.758027 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.758001 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-w0-3e2f96-kserve-6465bc6846-nvrsb"] Oct 30 17:18:00.793708 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.793680 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq"] Oct 30 17:18:00.795937 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.795917 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/distinct-2-w0-3e2f96-kserve-6b4bdc497c-hzjxq"] Oct 30 17:18:00.831351 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.831325 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj"] Oct 30 17:18:00.837341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.837251 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-3e2f96/simulator-w0-3e2f96-kserve-65b49dcc98-rzcgj"] Oct 30 17:18:00.965867 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.965835 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" path="/var/lib/kubelet/pods/32cf4325-2f94-455f-82b5-f33ad48c3884/volumes" Oct 30 17:18:00.966263 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.966250 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" path="/var/lib/kubelet/pods/6927cc23-033e-4bad-9a12-573afcf72cf0/volumes" Oct 30 17:18:00.966668 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:00.966641 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" path="/var/lib/kubelet/pods/6ae453e0-0c2c-4080-bef3-c2119549a4ef/volumes" Oct 30 17:18:07.005793 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:07.005754 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt"] Oct 30 17:18:07.019182 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:07.019153 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-3e2f96-ctqwt"] Oct 30 17:18:08.965687 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:08.965652 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4be1dda-22e4-41d5-9e00-835a86e0272d" path="/var/lib/kubelet/pods/e4be1dda-22e4-41d5-9e00-835a86e0272d/volumes" Oct 30 17:18:24.596703 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:18:24.596659 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:18:50.689550 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.689508 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w"] Oct 30 17:18:50.690094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.689961 6400 cpu_manager.go:414] "containerMap: removing container" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" Oct 30 17:18:50.690094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.689977 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" Oct 30 17:18:50.690094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.689995 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" Oct 30 17:18:50.690094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.690001 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" Oct 30 17:18:50.690094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.690014 6400 cpu_manager.go:414] "containerMap: removing container" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" Oct 30 17:18:50.690094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.690019 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" Oct 30 17:18:50.690094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.690031 6400 cpu_manager.go:414] "containerMap: removing container" podUID="fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256" containerName="revoke-keys" Oct 30 17:18:50.690094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.690036 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256" containerName="revoke-keys" Oct 30 17:18:50.690094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.690091 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6927cc23-033e-4bad-9a12-573afcf72cf0" containerName="main" Oct 30 17:18:50.690094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.690100 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256" containerName="revoke-keys" Oct 30 17:18:50.690420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.690111 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="32cf4325-2f94-455f-82b5-f33ad48c3884" containerName="main" Oct 30 17:18:50.690420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.690119 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ae453e0-0c2c-4080-bef3-c2119549a4ef" containerName="main" Oct 30 17:18:50.694805 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.694789 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:50.714056 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.714029 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w"] Oct 30 17:18:50.725479 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.725451 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:18:50.725669 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.725487 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:18:50.789531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.789486 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-hgt6w\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:50.789749 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.789640 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-hgt6w\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:50.789749 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.789692 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw96d\" (UniqueName: \"kubernetes.io/projected/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-kube-api-access-sw96d\") pod \"maas-api-revoke-sub-85237e45-hgt6w\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:50.890555 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.890506 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw96d\" (UniqueName: \"kubernetes.io/projected/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-kube-api-access-sw96d\") pod \"maas-api-revoke-sub-85237e45-hgt6w\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:50.890555 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.890581 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-hgt6w\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:50.890920 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.890654 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-hgt6w\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:50.891334 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.891308 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-hgt6w\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:50.893117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.893095 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-hgt6w\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:50.915313 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:50.915285 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw96d\" (UniqueName: \"kubernetes.io/projected/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-kube-api-access-sw96d\") pod \"maas-api-revoke-sub-85237e45-hgt6w\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:51.007313 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:51.007273 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:51.305168 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:51.305139 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w"] Oct 30 17:18:51.306940 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:18:51.306910 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd1ef2f_c1af_4aee_95e0_f8d54a177f5c.slice/crio-c1dd049d00a49f8efa221ec62fec2c3455beb6b19644c483b091cd7f2d4c31a9 WatchSource:0}: Error finding container c1dd049d00a49f8efa221ec62fec2c3455beb6b19644c483b091cd7f2d4c31a9: Status 404 returned error can't find the container with id c1dd049d00a49f8efa221ec62fec2c3455beb6b19644c483b091cd7f2d4c31a9 Oct 30 17:18:51.981784 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:51.981694 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" event={"ID":"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c","Type":"ContainerStarted","Data":"a72cba5f0ed38139f51e1a94f1d99fcd74470966834e053adc619cd12116d40f"} Oct 30 17:18:51.981784 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:51.981751 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" event={"ID":"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c","Type":"ContainerStarted","Data":"c1dd049d00a49f8efa221ec62fec2c3455beb6b19644c483b091cd7f2d4c31a9"} Oct 30 17:18:52.024170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:52.024121 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" podStartSLOduration=2.024106216 podStartE2EDuration="2.024106216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:52.023502593 +0000 UTC m=+3441.173783582" watchObservedRunningTime="2026-10-30 17:18:52.024106216 +0000 UTC m=+3441.174387203" Oct 30 17:18:52.987962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:52.987921 6400 generic.go:356] "Generic (PLEG): container finished" podID="8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c" containerID="a72cba5f0ed38139f51e1a94f1d99fcd74470966834e053adc619cd12116d40f" exitCode=0 Oct 30 17:18:52.988369 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:52.987999 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" event={"ID":"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c","Type":"ContainerDied","Data":"a72cba5f0ed38139f51e1a94f1d99fcd74470966834e053adc619cd12116d40f"} Oct 30 17:18:54.061779 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.061709 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9"] Oct 30 17:18:54.066164 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.066135 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.082589 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.082555 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-nztr5\"" Oct 30 17:18:54.097533 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.097503 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9"] Oct 30 17:18:54.120318 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.120290 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdb214-ef36-4692-b17c-e7296231f8a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.120442 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.120328 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.120442 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.120354 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.120558 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.120539 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.120637 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.120566 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9p8zr\" (UniqueName: \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-kube-api-access-9p8zr\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.120637 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.120593 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.120746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.120687 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.120746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.120709 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.120855 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.120748 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.134577 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.134556 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:54.221440 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.221412 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-openshift-ca\") pod \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " Oct 30 17:18:54.221647 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.221484 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-kube-api-access-sw96d\" (UniqueName: \"kubernetes.io/projected/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-kube-api-access-sw96d\") pod \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " Oct 30 17:18:54.221647 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.221518 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-xks-ca\") pod \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\" (UID: \"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c\") " Oct 30 17:18:54.221786 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.221764 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdb214-ef36-4692-b17c-e7296231f8a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.221847 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.221780 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-openshift-ca" pod "8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c" (UID: "8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:18:54.221847 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.221801 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.221954 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.221847 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.221954 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.221932 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.222067 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.221957 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9p8zr\" (UniqueName: \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-kube-api-access-9p8zr\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.222067 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.221989 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.222067 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.222041 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.222228 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.222066 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.222228 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.222129 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.222228 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.222176 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:54.222652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.222627 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.222874 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.222826 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdb214-ef36-4692-b17c-e7296231f8a2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.222874 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.222900 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.223185 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.223157 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.223438 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.223413 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.223952 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.223927 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-xks-ca" pod "8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c" (UID: "8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:18:54.224062 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.223949 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-kube-api-access-sw96d" pod "8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c" (UID: "8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c"). InnerVolumeSpecName "kube-api-access-sw96d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:18:54.224341 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.224322 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.224779 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.224762 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.247093 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.247029 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.247317 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.247300 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9p8zr\" (UniqueName: \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-kube-api-access-9p8zr\") pod \"maas-default-gateway-openshift-default-76f4876667-bndc9\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.323391 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.323339 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw96d\" (UniqueName: \"kubernetes.io/projected/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-kube-api-access-sw96d\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:54.323391 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.323384 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:18:54.381661 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.381634 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:54.596723 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:18:54.596635 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:18:54.698162 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.698138 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9"] Oct 30 17:18:54.698971 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:18:54.698942 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4fdb214_ef36_4692_b17c_e7296231f8a2.slice/crio-ae5d7954c9d95bd0f706e731e6119d15481e6ae21c5184339484aeaf3e61680a WatchSource:0}: Error finding container ae5d7954c9d95bd0f706e731e6119d15481e6ae21c5184339484aeaf3e61680a: Status 404 returned error can't find the container with id ae5d7954c9d95bd0f706e731e6119d15481e6ae21c5184339484aeaf3e61680a Oct 30 17:18:54.701202 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.701172 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:18:54.701284 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.701239 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:18:54.701284 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.701270 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:18:54.999444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.999413 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" Oct 30 17:18:54.999654 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.999458 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w" event={"ID":"8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c","Type":"ContainerDied","Data":"c1dd049d00a49f8efa221ec62fec2c3455beb6b19644c483b091cd7f2d4c31a9"} Oct 30 17:18:54.999654 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:54.999511 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1dd049d00a49f8efa221ec62fec2c3455beb6b19644c483b091cd7f2d4c31a9" Oct 30 17:18:55.001394 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:55.001364 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" event={"ID":"d4fdb214-ef36-4692-b17c-e7296231f8a2","Type":"ContainerStarted","Data":"3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a"} Oct 30 17:18:55.001634 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:55.001417 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" event={"ID":"d4fdb214-ef36-4692-b17c-e7296231f8a2","Type":"ContainerStarted","Data":"ae5d7954c9d95bd0f706e731e6119d15481e6ae21c5184339484aeaf3e61680a"} Oct 30 17:18:55.051108 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:55.051042 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" podStartSLOduration=1.051022607 podStartE2EDuration="1.051022607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:18:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:18:55.049818348 +0000 UTC m=+3444.200099335" watchObservedRunningTime="2026-10-30 17:18:55.051022607 +0000 UTC m=+3444.201303596" Oct 30 17:18:55.382171 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:55.382133 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:18:55.383854 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:55.383825 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bndc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 17:18:55.383999 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:55.383877 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 17:18:56.382048 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:56.382004 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bndc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 17:18:56.382338 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:56.382078 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 17:18:57.382313 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:57.382273 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bndc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 17:18:57.382530 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:57.382342 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 17:18:58.382720 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:58.382678 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bndc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 17:18:58.383160 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:58.382738 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 17:18:59.382307 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:59.382273 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bndc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 17:18:59.382490 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:18:59.382326 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 17:19:00.382560 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:00.382522 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bndc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 17:19:00.383040 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:00.382576 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 17:19:01.381989 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:01.381944 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-bndc9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body= Oct 30 17:19:01.382178 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:01.382004 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" Oct 30 17:19:02.967856 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:02.967824 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:19:02.968298 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:02.968232 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:19:02.969060 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:02.969037 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:19:09.118427 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.118382 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn"] Oct 30 17:19:09.119500 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.119474 6400 cpu_manager.go:414] "containerMap: removing container" podUID="8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c" containerName="revoke-keys" Oct 30 17:19:09.119660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.119646 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c" containerName="revoke-keys" Oct 30 17:19:09.119884 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.119873 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c" containerName="revoke-keys" Oct 30 17:19:09.123474 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.123454 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.154503 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.154474 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn"] Oct 30 17:19:09.176452 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.176417 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.176655 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.176470 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.176655 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.176546 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.176655 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.176621 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dfhw\" (UniqueName: \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-kube-api-access-2dfhw\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.176873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.176701 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.176873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.176742 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.176873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.176826 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.176873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.176855 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.177022 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.176891 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.278281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278239 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dfhw\" (UniqueName: \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-kube-api-access-2dfhw\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.278281 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278289 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.278590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278310 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.278590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278352 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.278590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278377 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.278590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278400 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.278590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278446 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.278590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278498 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.278590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278569 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.279036 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278851 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.279036 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.278896 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.279150 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.279056 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.279150 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.279090 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.279439 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.279366 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.281056 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.281034 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.281181 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.281154 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.303201 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.303171 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.303366 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.303342 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dfhw\" (UniqueName: \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-kube-api-access-2dfhw\") pod \"maas-default-gateway-openshift-default-76f4876667-qghkn\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.439491 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.439387 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:09.759070 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.759043 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn"] Oct 30 17:19:09.759706 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:19:09.759672 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a0a94a6_f67e_4fa4_9e2e_c1fb2e3cec63.slice/crio-16a48ffe16b6c91f461e8ec81427744a9f5cb70307ad9bdd95fd9b19cd62d681 WatchSource:0}: Error finding container 16a48ffe16b6c91f461e8ec81427744a9f5cb70307ad9bdd95fd9b19cd62d681: Status 404 returned error can't find the container with id 16a48ffe16b6c91f461e8ec81427744a9f5cb70307ad9bdd95fd9b19cd62d681 Oct 30 17:19:09.761809 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.761780 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:19:09.761873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.761844 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:19:09.761910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:09.761874 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:19:10.111721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:10.111627 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" event={"ID":"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63","Type":"ContainerStarted","Data":"c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e"} Oct 30 17:19:10.111721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:10.111686 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" event={"ID":"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63","Type":"ContainerStarted","Data":"16a48ffe16b6c91f461e8ec81427744a9f5cb70307ad9bdd95fd9b19cd62d681"} Oct 30 17:19:10.171996 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:10.171935 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podStartSLOduration=1.171915497 podStartE2EDuration="1.171915497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:19:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:19:10.171902783 +0000 UTC m=+3459.322183772" watchObservedRunningTime="2026-10-30 17:19:10.171915497 +0000 UTC m=+3459.322196486" Oct 30 17:19:10.440055 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:10.439936 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:10.441388 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:10.441364 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qghkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 17:19:10.441461 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:10.441412 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 17:19:11.440197 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:11.440157 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qghkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 17:19:11.440705 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:11.440218 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 17:19:12.440227 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:12.440190 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qghkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 17:19:12.440707 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:12.440261 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 17:19:13.440630 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:13.440562 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qghkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 17:19:13.441066 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:13.440638 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 17:19:14.440234 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:14.440196 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qghkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 17:19:14.440437 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:14.440257 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 17:19:15.440234 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:15.440191 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qghkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 17:19:15.440816 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:15.440263 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 17:19:16.439873 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:16.439837 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qghkn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body= Oct 30 17:19:16.440073 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:16.439918 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" Oct 30 17:19:18.006379 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:18.006344 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:18.006927 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:18.006806 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:18.007507 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:19:18.007489 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:19:24.596911 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:19:24.596868 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:19:54.597618 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:19:54.597564 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:20:09.272049 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.272010 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj"] Oct 30 17:20:09.275617 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.275576 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.303992 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.303951 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj"] Oct 30 17:20:09.440277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.440237 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.440511 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.440311 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74ef3e6-b26b-470a-abc1-6e80caa59558-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.440511 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.440449 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.440688 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.440528 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.440688 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.440648 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.440803 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.440711 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.440864 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.440806 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj2ft\" (UniqueName: \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-kube-api-access-pj2ft\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.440910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.440872 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.440910 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.440899 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.542415 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.542288 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.542415 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.542385 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.542725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.542420 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj2ft\" (UniqueName: \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-kube-api-access-pj2ft\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.542725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.542451 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.542725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.542481 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.542725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.542533 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.542725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.542627 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74ef3e6-b26b-470a-abc1-6e80caa59558-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.542725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.542663 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.542725 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.542715 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.544728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.543648 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.544728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.543888 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.544728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.544169 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.544728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.544672 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74ef3e6-b26b-470a-abc1-6e80caa59558-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.545072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.544771 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.548830 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.548771 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.561909 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.561876 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.575334 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.575271 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.576315 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.576284 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj2ft\" (UniqueName: \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-kube-api-access-pj2ft\") pod \"maas-default-gateway-openshift-default-76f4876667-4nbqj\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.594951 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.594929 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:09.957746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.957120 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj"] Oct 30 17:20:09.965636 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:20:09.965220 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda74ef3e6_b26b_470a_abc1_6e80caa59558.slice/crio-c6ab999741cd514f4bf9286e87103b42842e43abd0126126c457b3f202f15f23 WatchSource:0}: Error finding container c6ab999741cd514f4bf9286e87103b42842e43abd0126126c457b3f202f15f23: Status 404 returned error can't find the container with id c6ab999741cd514f4bf9286e87103b42842e43abd0126126c457b3f202f15f23 Oct 30 17:20:09.969303 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.968705 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:20:09.969303 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.968798 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:20:09.969303 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:09.968849 6400 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 17:20:10.481237 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:10.481195 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" event={"ID":"a74ef3e6-b26b-470a-abc1-6e80caa59558","Type":"ContainerStarted","Data":"af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576"} Oct 30 17:20:10.481718 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:10.481296 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" event={"ID":"a74ef3e6-b26b-470a-abc1-6e80caa59558","Type":"ContainerStarted","Data":"c6ab999741cd514f4bf9286e87103b42842e43abd0126126c457b3f202f15f23"} Oct 30 17:20:10.531733 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:10.531653 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podStartSLOduration=1.5316326230000001 podStartE2EDuration="1.531632623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:10.529249215 +0000 UTC m=+3519.679530205" watchObservedRunningTime="2026-10-30 17:20:10.531632623 +0000 UTC m=+3519.681913609" Oct 30 17:20:10.596072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:10.596045 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:11.597345 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:11.597308 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 17:20:11.597807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:11.597360 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 17:20:12.596415 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:12.596376 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 17:20:12.596579 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:12.596431 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 17:20:13.060913 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.060879 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95"] Oct 30 17:20:13.064377 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.064352 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.084903 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.084876 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95"] Oct 30 17:20:13.095656 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.095631 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:20:13.095826 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.095724 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:20:13.195285 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.195238 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-p2d95\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.195285 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.195292 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngcf9\" (UniqueName: \"kubernetes.io/projected/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-kube-api-access-ngcf9\") pod \"maas-api-revoke-sub-2e1f5b5f-p2d95\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.195568 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.195435 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-p2d95\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.296574 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.296530 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-p2d95\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.296838 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.296721 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-p2d95\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.296838 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.296753 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngcf9\" (UniqueName: \"kubernetes.io/projected/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-kube-api-access-ngcf9\") pod \"maas-api-revoke-sub-2e1f5b5f-p2d95\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.297376 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.297355 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-p2d95\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.299324 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.299299 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-p2d95\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.337176 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.334850 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngcf9\" (UniqueName: \"kubernetes.io/projected/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-kube-api-access-ngcf9\") pod \"maas-api-revoke-sub-2e1f5b5f-p2d95\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.377413 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.377375 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:13.595763 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.595673 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 17:20:13.595763 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.595729 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 17:20:13.697567 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:13.697535 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95"] Oct 30 17:20:13.698800 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:20:13.698767 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc61e65ad_d8c3_4786_b0ba_9ef9f30f05b4.slice/crio-c0de6908642c808e20bad18ad6166d64396637e8b065042498da37233cdbe71f WatchSource:0}: Error finding container c0de6908642c808e20bad18ad6166d64396637e8b065042498da37233cdbe71f: Status 404 returned error can't find the container with id c0de6908642c808e20bad18ad6166d64396637e8b065042498da37233cdbe71f Oct 30 17:20:14.534631 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:14.509852 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" event={"ID":"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4","Type":"ContainerStarted","Data":"96fb9fd2e0cecb5c66863cc2f77f1ea369a4e9b385a093370786609f2083b027"} Oct 30 17:20:14.535175 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:14.534743 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" event={"ID":"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4","Type":"ContainerStarted","Data":"c0de6908642c808e20bad18ad6166d64396637e8b065042498da37233cdbe71f"} Oct 30 17:20:14.575015 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:14.574961 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" podStartSLOduration=1.574948636 podStartE2EDuration="1.574948636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:14.573411414 +0000 UTC m=+3523.723692401" watchObservedRunningTime="2026-10-30 17:20:14.574948636 +0000 UTC m=+3523.725229637" Oct 30 17:20:14.595763 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:14.595728 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 17:20:14.595942 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:14.595783 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 17:20:15.539914 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:15.539877 6400 generic.go:356] "Generic (PLEG): container finished" podID="c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4" containerID="96fb9fd2e0cecb5c66863cc2f77f1ea369a4e9b385a093370786609f2083b027" exitCode=0 Oct 30 17:20:15.540334 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:15.539960 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" event={"ID":"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4","Type":"ContainerDied","Data":"96fb9fd2e0cecb5c66863cc2f77f1ea369a4e9b385a093370786609f2083b027"} Oct 30 17:20:15.595995 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:15.595900 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 17:20:15.595995 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:15.595952 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 17:20:16.596336 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.596304 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 17:20:16.596721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.596357 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 17:20:16.682635 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.682595 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:16.753656 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.753616 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-kube-api-access-ngcf9\" (UniqueName: \"kubernetes.io/projected/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-kube-api-access-ngcf9\") pod \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " Oct 30 17:20:16.753836 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.753761 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-openshift-ca\") pod \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " Oct 30 17:20:16.753836 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.753794 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-xks-ca\") pod \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\" (UID: \"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4\") " Oct 30 17:20:16.754201 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.754171 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-openshift-ca" pod "c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4" (UID: "c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:16.755996 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.755965 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-kube-api-access-ngcf9" pod "c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4" (UID: "c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4"). InnerVolumeSpecName "kube-api-access-ngcf9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:16.756076 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.756038 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-xks-ca" pod "c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4" (UID: "c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:20:16.855513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.855425 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngcf9\" (UniqueName: \"kubernetes.io/projected/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-kube-api-access-ngcf9\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.855513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.855456 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:20:16.855513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:16.855467 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:20:17.552034 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:17.552002 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" event={"ID":"c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4","Type":"ContainerDied","Data":"c0de6908642c808e20bad18ad6166d64396637e8b065042498da37233cdbe71f"} Oct 30 17:20:17.552222 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:17.552041 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95" Oct 30 17:20:17.552222 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:17.552050 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0de6908642c808e20bad18ad6166d64396637e8b065042498da37233cdbe71f" Oct 30 17:20:17.595465 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:17.595421 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbqj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body= Oct 30 17:20:17.595678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:17.595496 6400 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" Oct 30 17:20:18.603307 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:18.603275 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:18.603864 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:18.603637 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:18.604571 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:18.604548 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:20:24.271242 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.271207 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8"] Oct 30 17:20:24.271659 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.271580 6400 cpu_manager.go:414] "containerMap: removing container" podUID="c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4" containerName="revoke-keys" Oct 30 17:20:24.271659 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.271591 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4" containerName="revoke-keys" Oct 30 17:20:24.271756 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.271697 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4" containerName="revoke-keys" Oct 30 17:20:24.274770 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.274749 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.295951 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.295925 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8"] Oct 30 17:20:24.305515 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.305490 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:20:24.305515 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.305512 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:20:24.330026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.329992 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-k7tx8\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.330151 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.330042 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmht7\" (UniqueName: \"kubernetes.io/projected/e29d7aee-74d8-472f-8b41-87720fb09c09-kube-api-access-qmht7\") pod \"maas-api-revoke-sub-ec843018-k7tx8\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.330193 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.330175 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-k7tx8\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.430660 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.430627 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-k7tx8\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.430828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.430696 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-k7tx8\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.430828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.430729 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmht7\" (UniqueName: \"kubernetes.io/projected/e29d7aee-74d8-472f-8b41-87720fb09c09-kube-api-access-qmht7\") pod \"maas-api-revoke-sub-ec843018-k7tx8\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.431352 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.431275 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-k7tx8\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.432841 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.432821 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-k7tx8\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.455363 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.455334 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmht7\" (UniqueName: \"kubernetes.io/projected/e29d7aee-74d8-472f-8b41-87720fb09c09-kube-api-access-qmht7\") pod \"maas-api-revoke-sub-ec843018-k7tx8\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.592976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.592901 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:24.600773 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:20:24.600744 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:20:24.894990 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:20:24.894961 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode29d7aee_74d8_472f_8b41_87720fb09c09.slice/crio-ad36c029517bab2d197bf9ee7b5514e85d1d26e621cb2f41e36a2fd97d8bb374 WatchSource:0}: Error finding container ad36c029517bab2d197bf9ee7b5514e85d1d26e621cb2f41e36a2fd97d8bb374: Status 404 returned error can't find the container with id ad36c029517bab2d197bf9ee7b5514e85d1d26e621cb2f41e36a2fd97d8bb374 Oct 30 17:20:24.895672 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:24.895638 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8"] Oct 30 17:20:25.597220 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:25.597182 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" event={"ID":"e29d7aee-74d8-472f-8b41-87720fb09c09","Type":"ContainerStarted","Data":"f3635933b5c451f10860da921ad093b65c8ac23f722ede7884aeeaa422a97476"} Oct 30 17:20:25.597744 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:25.597236 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" event={"ID":"e29d7aee-74d8-472f-8b41-87720fb09c09","Type":"ContainerStarted","Data":"ad36c029517bab2d197bf9ee7b5514e85d1d26e621cb2f41e36a2fd97d8bb374"} Oct 30 17:20:25.640274 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:25.640220 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" podStartSLOduration=1.640205558 podStartE2EDuration="1.640205558s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:20:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:20:25.639677661 +0000 UTC m=+3534.789958651" watchObservedRunningTime="2026-10-30 17:20:25.640205558 +0000 UTC m=+3534.790486545" Oct 30 17:20:26.604250 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:26.604215 6400 generic.go:356] "Generic (PLEG): container finished" podID="e29d7aee-74d8-472f-8b41-87720fb09c09" containerID="f3635933b5c451f10860da921ad093b65c8ac23f722ede7884aeeaa422a97476" exitCode=0 Oct 30 17:20:26.604697 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:26.604270 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" event={"ID":"e29d7aee-74d8-472f-8b41-87720fb09c09","Type":"ContainerDied","Data":"f3635933b5c451f10860da921ad093b65c8ac23f722ede7884aeeaa422a97476"} Oct 30 17:20:27.743460 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:27.743434 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:27.864934 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:27.864854 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-openshift-ca\") pod \"e29d7aee-74d8-472f-8b41-87720fb09c09\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " Oct 30 17:20:27.865093 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:27.864955 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-xks-ca\") pod \"e29d7aee-74d8-472f-8b41-87720fb09c09\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " Oct 30 17:20:27.865093 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:27.864986 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e29d7aee-74d8-472f-8b41-87720fb09c09-kube-api-access-qmht7\" (UniqueName: \"kubernetes.io/projected/e29d7aee-74d8-472f-8b41-87720fb09c09-kube-api-access-qmht7\") pod \"e29d7aee-74d8-472f-8b41-87720fb09c09\" (UID: \"e29d7aee-74d8-472f-8b41-87720fb09c09\") " Oct 30 17:20:27.865264 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:27.865238 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-openshift-ca" pod "e29d7aee-74d8-472f-8b41-87720fb09c09" (UID: "e29d7aee-74d8-472f-8b41-87720fb09c09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:20:27.867190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:27.867167 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-xks-ca" pod "e29d7aee-74d8-472f-8b41-87720fb09c09" (UID: "e29d7aee-74d8-472f-8b41-87720fb09c09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:20:27.867190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:27.867184 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e29d7aee-74d8-472f-8b41-87720fb09c09-kube-api-access-qmht7" pod "e29d7aee-74d8-472f-8b41-87720fb09c09" (UID: "e29d7aee-74d8-472f-8b41-87720fb09c09"). InnerVolumeSpecName "kube-api-access-qmht7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:20:27.965849 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:27.965814 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:20:27.965849 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:27.965843 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmht7\" (UniqueName: \"kubernetes.io/projected/e29d7aee-74d8-472f-8b41-87720fb09c09-kube-api-access-qmht7\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:20:27.965849 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:27.965854 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e29d7aee-74d8-472f-8b41-87720fb09c09-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:20:28.617044 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:28.617015 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" Oct 30 17:20:28.617235 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:28.617064 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8" event={"ID":"e29d7aee-74d8-472f-8b41-87720fb09c09","Type":"ContainerDied","Data":"ad36c029517bab2d197bf9ee7b5514e85d1d26e621cb2f41e36a2fd97d8bb374"} Oct 30 17:20:28.617235 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:20:28.617120 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad36c029517bab2d197bf9ee7b5514e85d1d26e621cb2f41e36a2fd97d8bb374" Oct 30 17:20:54.597001 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:20:54.596961 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:21:12.186420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.186385 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd"] Oct 30 17:21:12.186992 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.186841 6400 cpu_manager.go:414] "containerMap: removing container" podUID="e29d7aee-74d8-472f-8b41-87720fb09c09" containerName="revoke-keys" Oct 30 17:21:12.186992 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.186853 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e29d7aee-74d8-472f-8b41-87720fb09c09" containerName="revoke-keys" Oct 30 17:21:12.186992 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.186921 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="e29d7aee-74d8-472f-8b41-87720fb09c09" containerName="revoke-keys" Oct 30 17:21:12.189280 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.189262 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.210248 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.210221 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd"] Oct 30 17:21:12.221211 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.221185 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:21:12.221339 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.221212 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:21:12.271072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.271029 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwmtj\" (UniqueName: \"kubernetes.io/projected/677cc0e1-27a8-48b7-9986-913d69d2bb10-kube-api-access-wwmtj\") pod \"maas-api-revoke-sub-046e1343-nrrvd\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.271269 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.271156 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-nrrvd\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.271269 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.271206 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-nrrvd\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.372453 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.372409 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-nrrvd\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.372678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.372474 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-nrrvd\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.372678 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.372533 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwmtj\" (UniqueName: \"kubernetes.io/projected/677cc0e1-27a8-48b7-9986-913d69d2bb10-kube-api-access-wwmtj\") pod \"maas-api-revoke-sub-046e1343-nrrvd\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.373126 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.373106 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-nrrvd\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.375169 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.375151 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-nrrvd\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.397721 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.397684 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwmtj\" (UniqueName: \"kubernetes.io/projected/677cc0e1-27a8-48b7-9986-913d69d2bb10-kube-api-access-wwmtj\") pod \"maas-api-revoke-sub-046e1343-nrrvd\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.500837 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.500803 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:12.799241 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.799213 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd"] Oct 30 17:21:12.800524 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:21:12.800500 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod677cc0e1_27a8_48b7_9986_913d69d2bb10.slice/crio-2ea9bfc0d9327090c1195f7046d6b49ddb8d955e4e26a98aefc85abcdaaee6aa WatchSource:0}: Error finding container 2ea9bfc0d9327090c1195f7046d6b49ddb8d955e4e26a98aefc85abcdaaee6aa: Status 404 returned error can't find the container with id 2ea9bfc0d9327090c1195f7046d6b49ddb8d955e4e26a98aefc85abcdaaee6aa Oct 30 17:21:12.887528 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:12.887491 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" event={"ID":"677cc0e1-27a8-48b7-9986-913d69d2bb10","Type":"ContainerStarted","Data":"2ea9bfc0d9327090c1195f7046d6b49ddb8d955e4e26a98aefc85abcdaaee6aa"} Oct 30 17:21:13.894197 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:13.894112 6400 generic.go:356] "Generic (PLEG): container finished" podID="677cc0e1-27a8-48b7-9986-913d69d2bb10" containerID="83efc4c5fa17481b3f44cd34d922484ee3011f775be152fcab5744589097047b" exitCode=0 Oct 30 17:21:13.894197 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:13.894164 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" event={"ID":"677cc0e1-27a8-48b7-9986-913d69d2bb10","Type":"ContainerDied","Data":"83efc4c5fa17481b3f44cd34d922484ee3011f775be152fcab5744589097047b"} Oct 30 17:21:15.039753 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.039722 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:15.098018 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.097986 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-openshift-ca\") pod \"677cc0e1-27a8-48b7-9986-913d69d2bb10\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " Oct 30 17:21:15.098190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.098119 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/677cc0e1-27a8-48b7-9986-913d69d2bb10-kube-api-access-wwmtj\" (UniqueName: \"kubernetes.io/projected/677cc0e1-27a8-48b7-9986-913d69d2bb10-kube-api-access-wwmtj\") pod \"677cc0e1-27a8-48b7-9986-913d69d2bb10\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " Oct 30 17:21:15.098190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.098157 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-xks-ca\") pod \"677cc0e1-27a8-48b7-9986-913d69d2bb10\" (UID: \"677cc0e1-27a8-48b7-9986-913d69d2bb10\") " Oct 30 17:21:15.098385 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.098360 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-openshift-ca" pod "677cc0e1-27a8-48b7-9986-913d69d2bb10" (UID: "677cc0e1-27a8-48b7-9986-913d69d2bb10"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:15.098548 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.098525 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:21:15.100399 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.100376 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/677cc0e1-27a8-48b7-9986-913d69d2bb10-kube-api-access-wwmtj" pod "677cc0e1-27a8-48b7-9986-913d69d2bb10" (UID: "677cc0e1-27a8-48b7-9986-913d69d2bb10"). InnerVolumeSpecName "kube-api-access-wwmtj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:15.100399 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.100392 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-xks-ca" pod "677cc0e1-27a8-48b7-9986-913d69d2bb10" (UID: "677cc0e1-27a8-48b7-9986-913d69d2bb10"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:15.199210 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.199131 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwmtj\" (UniqueName: \"kubernetes.io/projected/677cc0e1-27a8-48b7-9986-913d69d2bb10-kube-api-access-wwmtj\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:21:15.199210 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.199158 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/677cc0e1-27a8-48b7-9986-913d69d2bb10-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:21:15.906853 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.906816 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" event={"ID":"677cc0e1-27a8-48b7-9986-913d69d2bb10","Type":"ContainerDied","Data":"2ea9bfc0d9327090c1195f7046d6b49ddb8d955e4e26a98aefc85abcdaaee6aa"} Oct 30 17:21:15.906853 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.906841 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd" Oct 30 17:21:15.907097 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:15.906889 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2ea9bfc0d9327090c1195f7046d6b49ddb8d955e4e26a98aefc85abcdaaee6aa" Oct 30 17:21:22.490180 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.490150 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr"] Oct 30 17:21:22.490616 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.490513 6400 cpu_manager.go:414] "containerMap: removing container" podUID="677cc0e1-27a8-48b7-9986-913d69d2bb10" containerName="revoke-keys" Oct 30 17:21:22.490616 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.490524 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="677cc0e1-27a8-48b7-9986-913d69d2bb10" containerName="revoke-keys" Oct 30 17:21:22.490709 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.490620 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="677cc0e1-27a8-48b7-9986-913d69d2bb10" containerName="revoke-keys" Oct 30 17:21:22.494791 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.494761 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:22.514193 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.514162 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr"] Oct 30 17:21:22.525130 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.525102 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:21:22.525286 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.525172 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:21:22.571146 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.571116 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-788wr\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:22.571298 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.571234 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cc9gr\" (UniqueName: \"kubernetes.io/projected/d568734f-189b-4a9b-976f-efdb2fdd844c-kube-api-access-cc9gr\") pod \"maas-api-revoke-sub-d3fba056-788wr\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:22.571298 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.571273 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-788wr\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:22.672387 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.672356 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-788wr\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:22.672538 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.672432 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cc9gr\" (UniqueName: \"kubernetes.io/projected/d568734f-189b-4a9b-976f-efdb2fdd844c-kube-api-access-cc9gr\") pod \"maas-api-revoke-sub-d3fba056-788wr\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:22.672538 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.672468 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-788wr\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:22.673103 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.673076 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-788wr\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:22.674836 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.674812 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-788wr\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:22.697154 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.697135 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cc9gr\" (UniqueName: \"kubernetes.io/projected/d568734f-189b-4a9b-976f-efdb2fdd844c-kube-api-access-cc9gr\") pod \"maas-api-revoke-sub-d3fba056-788wr\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:22.809451 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:22.809378 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:23.112751 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:23.112725 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr"] Oct 30 17:21:23.113201 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:21:23.113175 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd568734f_189b_4a9b_976f_efdb2fdd844c.slice/crio-df3cf6a5c4731151070737b31871175dc69d02438b56c18cc397c13e3155864c WatchSource:0}: Error finding container df3cf6a5c4731151070737b31871175dc69d02438b56c18cc397c13e3155864c: Status 404 returned error can't find the container with id df3cf6a5c4731151070737b31871175dc69d02438b56c18cc397c13e3155864c Oct 30 17:21:23.951304 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:23.950952 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" event={"ID":"d568734f-189b-4a9b-976f-efdb2fdd844c","Type":"ContainerStarted","Data":"06bb68969718694b27f24c82104e9fcbad323e98422d63a2923c4695f236d0e6"} Oct 30 17:21:23.951304 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:23.951017 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" event={"ID":"d568734f-189b-4a9b-976f-efdb2fdd844c","Type":"ContainerStarted","Data":"df3cf6a5c4731151070737b31871175dc69d02438b56c18cc397c13e3155864c"} Oct 30 17:21:23.993795 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:23.993738 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" podStartSLOduration=1.9937211989999999 podStartE2EDuration="1.993721199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:23.993356361 +0000 UTC m=+3593.143637348" watchObservedRunningTime="2026-10-30 17:21:23.993721199 +0000 UTC m=+3593.144002189" Oct 30 17:21:24.620155 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:21:24.620073 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:21:24.978738 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:24.978652 6400 generic.go:356] "Generic (PLEG): container finished" podID="d568734f-189b-4a9b-976f-efdb2fdd844c" containerID="06bb68969718694b27f24c82104e9fcbad323e98422d63a2923c4695f236d0e6" exitCode=0 Oct 30 17:21:24.988983 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:24.988948 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" event={"ID":"d568734f-189b-4a9b-976f-efdb2fdd844c","Type":"ContainerDied","Data":"06bb68969718694b27f24c82104e9fcbad323e98422d63a2923c4695f236d0e6"} Oct 30 17:21:26.264760 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.264687 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:26.344432 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.344405 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-xks-ca\") pod \"d568734f-189b-4a9b-976f-efdb2fdd844c\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " Oct 30 17:21:26.344636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.344479 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-openshift-ca\") pod \"d568734f-189b-4a9b-976f-efdb2fdd844c\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " Oct 30 17:21:26.344636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.344536 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d568734f-189b-4a9b-976f-efdb2fdd844c-kube-api-access-cc9gr\" (UniqueName: \"kubernetes.io/projected/d568734f-189b-4a9b-976f-efdb2fdd844c-kube-api-access-cc9gr\") pod \"d568734f-189b-4a9b-976f-efdb2fdd844c\" (UID: \"d568734f-189b-4a9b-976f-efdb2fdd844c\") " Oct 30 17:21:26.344922 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.344900 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-openshift-ca" pod "d568734f-189b-4a9b-976f-efdb2fdd844c" (UID: "d568734f-189b-4a9b-976f-efdb2fdd844c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:26.346816 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.346793 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-xks-ca" pod "d568734f-189b-4a9b-976f-efdb2fdd844c" (UID: "d568734f-189b-4a9b-976f-efdb2fdd844c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:26.346904 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.346808 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d568734f-189b-4a9b-976f-efdb2fdd844c-kube-api-access-cc9gr" pod "d568734f-189b-4a9b-976f-efdb2fdd844c" (UID: "d568734f-189b-4a9b-976f-efdb2fdd844c"). InnerVolumeSpecName "kube-api-access-cc9gr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:26.445593 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.445562 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:21:26.445593 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.445591 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cc9gr\" (UniqueName: \"kubernetes.io/projected/d568734f-189b-4a9b-976f-efdb2fdd844c-kube-api-access-cc9gr\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:21:26.445593 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.445620 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d568734f-189b-4a9b-976f-efdb2fdd844c-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:21:26.995125 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.995094 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" event={"ID":"d568734f-189b-4a9b-976f-efdb2fdd844c","Type":"ContainerDied","Data":"df3cf6a5c4731151070737b31871175dc69d02438b56c18cc397c13e3155864c"} Oct 30 17:21:26.995270 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.995141 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df3cf6a5c4731151070737b31871175dc69d02438b56c18cc397c13e3155864c" Oct 30 17:21:26.995270 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:26.995103 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr" Oct 30 17:21:32.724559 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:32.724526 6400 scope.go:122] "RemoveContainer" containerID="dd8c23379d6f83137bd0b30ace1f3d9862d6928516550f97b798e0c89cd060ca" Oct 30 17:21:32.733623 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:32.733589 6400 scope.go:122] "RemoveContainer" containerID="b58492d331aaf3304dbdbc0774b144fef2cb11da4f56f1959180b51d56b04b1f" Oct 30 17:21:44.946626 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:44.946577 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr"] Oct 30 17:21:44.948934 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:44.946964 6400 cpu_manager.go:414] "containerMap: removing container" podUID="d568734f-189b-4a9b-976f-efdb2fdd844c" containerName="revoke-keys" Oct 30 17:21:44.948934 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:44.946975 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d568734f-189b-4a9b-976f-efdb2fdd844c" containerName="revoke-keys" Oct 30 17:21:44.948934 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:44.947059 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d568734f-189b-4a9b-976f-efdb2fdd844c" containerName="revoke-keys" Oct 30 17:21:44.949963 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:44.949947 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:44.970643 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:44.970588 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr"] Oct 30 17:21:44.980882 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:44.980860 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:21:44.981008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:44.980942 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:21:45.036203 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.036175 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8tssr\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:45.036347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.036215 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpbhf\" (UniqueName: \"kubernetes.io/projected/ee36df14-bf62-4539-b242-0993c87d7011-kube-api-access-wpbhf\") pod \"maas-api-revoke-sub-ae594979-8tssr\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:45.036347 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.036328 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8tssr\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:45.136851 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.136820 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8tssr\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:45.137018 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.136872 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpbhf\" (UniqueName: \"kubernetes.io/projected/ee36df14-bf62-4539-b242-0993c87d7011-kube-api-access-wpbhf\") pod \"maas-api-revoke-sub-ae594979-8tssr\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:45.137018 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.136905 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8tssr\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:45.137590 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.137571 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-8tssr\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:45.139338 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.139316 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-8tssr\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:45.161797 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.161769 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpbhf\" (UniqueName: \"kubernetes.io/projected/ee36df14-bf62-4539-b242-0993c87d7011-kube-api-access-wpbhf\") pod \"maas-api-revoke-sub-ae594979-8tssr\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:45.262401 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.262370 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:45.563829 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:45.563801 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr"] Oct 30 17:21:45.564477 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:21:45.564452 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee36df14_bf62_4539_b242_0993c87d7011.slice/crio-3498b660813b3fd7076e5ff61506aed071f2f53161319ab6b1e08be210006ca4 WatchSource:0}: Error finding container 3498b660813b3fd7076e5ff61506aed071f2f53161319ab6b1e08be210006ca4: Status 404 returned error can't find the container with id 3498b660813b3fd7076e5ff61506aed071f2f53161319ab6b1e08be210006ca4 Oct 30 17:21:46.127260 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:46.127228 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" event={"ID":"ee36df14-bf62-4539-b242-0993c87d7011","Type":"ContainerStarted","Data":"5aac1a1324a492d4e4d883b563a81ce2d87771864ebdd5e1e700d9f2938d89a0"} Oct 30 17:21:46.127989 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:46.127323 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" event={"ID":"ee36df14-bf62-4539-b242-0993c87d7011","Type":"ContainerStarted","Data":"3498b660813b3fd7076e5ff61506aed071f2f53161319ab6b1e08be210006ca4"} Oct 30 17:21:46.170122 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:46.170077 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" podStartSLOduration=2.170065388 podStartE2EDuration="2.170065388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:21:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:21:46.16919321 +0000 UTC m=+3615.319474198" watchObservedRunningTime="2026-10-30 17:21:46.170065388 +0000 UTC m=+3615.320346414" Oct 30 17:21:47.133615 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:47.133568 6400 generic.go:356] "Generic (PLEG): container finished" podID="ee36df14-bf62-4539-b242-0993c87d7011" containerID="5aac1a1324a492d4e4d883b563a81ce2d87771864ebdd5e1e700d9f2938d89a0" exitCode=0 Oct 30 17:21:47.134036 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:47.133655 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" event={"ID":"ee36df14-bf62-4539-b242-0993c87d7011","Type":"ContainerDied","Data":"5aac1a1324a492d4e4d883b563a81ce2d87771864ebdd5e1e700d9f2938d89a0"} Oct 30 17:21:48.274336 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:48.274312 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:48.366012 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:48.365981 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-xks-ca\") pod \"ee36df14-bf62-4539-b242-0993c87d7011\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " Oct 30 17:21:48.366193 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:48.366080 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-openshift-ca\") pod \"ee36df14-bf62-4539-b242-0993c87d7011\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " Oct 30 17:21:48.366193 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:48.366130 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee36df14-bf62-4539-b242-0993c87d7011-kube-api-access-wpbhf\" (UniqueName: \"kubernetes.io/projected/ee36df14-bf62-4539-b242-0993c87d7011-kube-api-access-wpbhf\") pod \"ee36df14-bf62-4539-b242-0993c87d7011\" (UID: \"ee36df14-bf62-4539-b242-0993c87d7011\") " Oct 30 17:21:48.366421 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:48.366399 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-openshift-ca" pod "ee36df14-bf62-4539-b242-0993c87d7011" (UID: "ee36df14-bf62-4539-b242-0993c87d7011"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:21:48.366505 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:48.366465 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:21:48.368369 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:48.368347 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-xks-ca" pod "ee36df14-bf62-4539-b242-0993c87d7011" (UID: "ee36df14-bf62-4539-b242-0993c87d7011"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:21:48.368453 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:48.368379 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee36df14-bf62-4539-b242-0993c87d7011-kube-api-access-wpbhf" pod "ee36df14-bf62-4539-b242-0993c87d7011" (UID: "ee36df14-bf62-4539-b242-0993c87d7011"). InnerVolumeSpecName "kube-api-access-wpbhf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:21:48.467657 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:48.467583 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ee36df14-bf62-4539-b242-0993c87d7011-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:21:48.467657 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:48.467618 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpbhf\" (UniqueName: \"kubernetes.io/projected/ee36df14-bf62-4539-b242-0993c87d7011-kube-api-access-wpbhf\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:21:49.145350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:49.145319 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" Oct 30 17:21:49.145350 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:49.145325 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr" event={"ID":"ee36df14-bf62-4539-b242-0993c87d7011","Type":"ContainerDied","Data":"3498b660813b3fd7076e5ff61506aed071f2f53161319ab6b1e08be210006ca4"} Oct 30 17:21:49.145555 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:21:49.145375 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3498b660813b3fd7076e5ff61506aed071f2f53161319ab6b1e08be210006ca4" Oct 30 17:21:54.597685 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:21:54.597649 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:22:06.496782 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.496748 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp"] Oct 30 17:22:06.497330 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.497241 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ee36df14-bf62-4539-b242-0993c87d7011" containerName="revoke-keys" Oct 30 17:22:06.497330 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.497261 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee36df14-bf62-4539-b242-0993c87d7011" containerName="revoke-keys" Oct 30 17:22:06.497449 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.497423 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee36df14-bf62-4539-b242-0993c87d7011" containerName="revoke-keys" Oct 30 17:22:06.500554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.500533 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:06.521076 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.521045 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp"] Oct 30 17:22:06.531094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.531070 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:22:06.531245 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.531124 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:22:06.539629 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.539586 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t589\" (UniqueName: \"kubernetes.io/projected/ca133507-3790-404c-bb3d-1e631b7048dc-kube-api-access-8t589\") pod \"maas-api-revoke-sub-a4aef7d7-tr5hp\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:06.539774 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.539683 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tr5hp\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:06.539856 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.539822 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tr5hp\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:06.640450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.640416 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t589\" (UniqueName: \"kubernetes.io/projected/ca133507-3790-404c-bb3d-1e631b7048dc-kube-api-access-8t589\") pod \"maas-api-revoke-sub-a4aef7d7-tr5hp\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:06.640450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.640455 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tr5hp\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:06.640686 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.640500 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tr5hp\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:06.641145 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.641114 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tr5hp\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:06.643070 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.643046 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-tr5hp\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:06.664530 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.664507 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t589\" (UniqueName: \"kubernetes.io/projected/ca133507-3790-404c-bb3d-1e631b7048dc-kube-api-access-8t589\") pod \"maas-api-revoke-sub-a4aef7d7-tr5hp\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:06.814132 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:06.814047 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:07.111818 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:07.111794 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp"] Oct 30 17:22:07.112382 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:22:07.112357 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca133507_3790_404c_bb3d_1e631b7048dc.slice/crio-f2adb7eb4559de82c20068355dc49fd568cbe7df50c9a54bb06e5411a6dc27d7 WatchSource:0}: Error finding container f2adb7eb4559de82c20068355dc49fd568cbe7df50c9a54bb06e5411a6dc27d7: Status 404 returned error can't find the container with id f2adb7eb4559de82c20068355dc49fd568cbe7df50c9a54bb06e5411a6dc27d7 Oct 30 17:22:07.266526 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:07.266496 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" event={"ID":"ca133507-3790-404c-bb3d-1e631b7048dc","Type":"ContainerStarted","Data":"f2adb7eb4559de82c20068355dc49fd568cbe7df50c9a54bb06e5411a6dc27d7"} Oct 30 17:22:08.273748 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:08.273713 6400 generic.go:356] "Generic (PLEG): container finished" podID="ca133507-3790-404c-bb3d-1e631b7048dc" containerID="544edae1c1b3936497846995e63175361357b867e177b23f617651ba239dd0de" exitCode=0 Oct 30 17:22:08.274151 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:08.273800 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" event={"ID":"ca133507-3790-404c-bb3d-1e631b7048dc","Type":"ContainerDied","Data":"544edae1c1b3936497846995e63175361357b867e177b23f617651ba239dd0de"} Oct 30 17:22:09.414576 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:09.414550 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:09.467837 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:09.467806 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-xks-ca\") pod \"ca133507-3790-404c-bb3d-1e631b7048dc\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " Oct 30 17:22:09.467987 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:09.467858 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-openshift-ca\") pod \"ca133507-3790-404c-bb3d-1e631b7048dc\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " Oct 30 17:22:09.467987 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:09.467909 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca133507-3790-404c-bb3d-1e631b7048dc-kube-api-access-8t589\" (UniqueName: \"kubernetes.io/projected/ca133507-3790-404c-bb3d-1e631b7048dc-kube-api-access-8t589\") pod \"ca133507-3790-404c-bb3d-1e631b7048dc\" (UID: \"ca133507-3790-404c-bb3d-1e631b7048dc\") " Oct 30 17:22:09.468233 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:09.468210 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-openshift-ca" pod "ca133507-3790-404c-bb3d-1e631b7048dc" (UID: "ca133507-3790-404c-bb3d-1e631b7048dc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:09.470202 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:09.470171 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-xks-ca" pod "ca133507-3790-404c-bb3d-1e631b7048dc" (UID: "ca133507-3790-404c-bb3d-1e631b7048dc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:09.470315 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:09.470197 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca133507-3790-404c-bb3d-1e631b7048dc-kube-api-access-8t589" pod "ca133507-3790-404c-bb3d-1e631b7048dc" (UID: "ca133507-3790-404c-bb3d-1e631b7048dc"). InnerVolumeSpecName "kube-api-access-8t589". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:09.569744 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:09.569662 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:22:09.569744 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:09.569688 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca133507-3790-404c-bb3d-1e631b7048dc-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:22:09.569744 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:09.569699 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8t589\" (UniqueName: \"kubernetes.io/projected/ca133507-3790-404c-bb3d-1e631b7048dc-kube-api-access-8t589\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:22:10.286264 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:10.286234 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" Oct 30 17:22:10.286449 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:10.286268 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp" event={"ID":"ca133507-3790-404c-bb3d-1e631b7048dc","Type":"ContainerDied","Data":"f2adb7eb4559de82c20068355dc49fd568cbe7df50c9a54bb06e5411a6dc27d7"} Oct 30 17:22:10.286449 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:10.286315 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2adb7eb4559de82c20068355dc49fd568cbe7df50c9a54bb06e5411a6dc27d7" Oct 30 17:22:24.606241 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:22:24.606208 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:22:26.768537 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.768502 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb"] Oct 30 17:22:26.769144 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.769125 6400 cpu_manager.go:414] "containerMap: removing container" podUID="ca133507-3790-404c-bb3d-1e631b7048dc" containerName="revoke-keys" Oct 30 17:22:26.769195 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.769148 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca133507-3790-404c-bb3d-1e631b7048dc" containerName="revoke-keys" Oct 30 17:22:26.769268 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.769255 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca133507-3790-404c-bb3d-1e631b7048dc" containerName="revoke-keys" Oct 30 17:22:26.773205 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.773187 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:26.790973 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.790938 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb"] Oct 30 17:22:26.801091 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.801063 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:22:26.801214 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.801104 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:22:26.845196 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.845163 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wh7gq\" (UniqueName: \"kubernetes.io/projected/f6657ce8-33e5-4197-9881-f89bb40636b5-kube-api-access-wh7gq\") pod \"maas-api-revoke-sub-79805ca9-mwmrb\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:26.845359 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.845321 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mwmrb\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:26.845427 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.845405 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mwmrb\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:26.946720 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.946685 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wh7gq\" (UniqueName: \"kubernetes.io/projected/f6657ce8-33e5-4197-9881-f89bb40636b5-kube-api-access-wh7gq\") pod \"maas-api-revoke-sub-79805ca9-mwmrb\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:26.946948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.946783 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mwmrb\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:26.946948 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.946813 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mwmrb\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:26.947427 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.947403 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-mwmrb\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:26.949349 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.949328 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-mwmrb\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:26.970909 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:26.970889 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wh7gq\" (UniqueName: \"kubernetes.io/projected/f6657ce8-33e5-4197-9881-f89bb40636b5-kube-api-access-wh7gq\") pod \"maas-api-revoke-sub-79805ca9-mwmrb\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:27.089033 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:27.088964 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:27.386731 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:27.386702 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb"] Oct 30 17:22:27.387475 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:22:27.387446 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6657ce8_33e5_4197_9881_f89bb40636b5.slice/crio-6d7f7b8dcb6c6519aa7a0792f74d6ff1fe2cb3b43f920792f874cefdc6dc327f WatchSource:0}: Error finding container 6d7f7b8dcb6c6519aa7a0792f74d6ff1fe2cb3b43f920792f874cefdc6dc327f: Status 404 returned error can't find the container with id 6d7f7b8dcb6c6519aa7a0792f74d6ff1fe2cb3b43f920792f874cefdc6dc327f Oct 30 17:22:27.397330 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:27.397302 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" event={"ID":"f6657ce8-33e5-4197-9881-f89bb40636b5","Type":"ContainerStarted","Data":"6d7f7b8dcb6c6519aa7a0792f74d6ff1fe2cb3b43f920792f874cefdc6dc327f"} Oct 30 17:22:28.404793 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:28.404758 6400 generic.go:356] "Generic (PLEG): container finished" podID="f6657ce8-33e5-4197-9881-f89bb40636b5" containerID="e6f90858427133ce644bf0fc79842d38174998c8725c22c753ab2086bc486ae9" exitCode=0 Oct 30 17:22:28.405164 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:28.404800 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" event={"ID":"f6657ce8-33e5-4197-9881-f89bb40636b5","Type":"ContainerDied","Data":"e6f90858427133ce644bf0fc79842d38174998c8725c22c753ab2086bc486ae9"} Oct 30 17:22:29.547964 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:29.547941 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:29.673552 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:29.673477 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-xks-ca\") pod \"f6657ce8-33e5-4197-9881-f89bb40636b5\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " Oct 30 17:22:29.673552 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:29.673526 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-openshift-ca\") pod \"f6657ce8-33e5-4197-9881-f89bb40636b5\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " Oct 30 17:22:29.673795 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:29.673646 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f6657ce8-33e5-4197-9881-f89bb40636b5-kube-api-access-wh7gq\" (UniqueName: \"kubernetes.io/projected/f6657ce8-33e5-4197-9881-f89bb40636b5-kube-api-access-wh7gq\") pod \"f6657ce8-33e5-4197-9881-f89bb40636b5\" (UID: \"f6657ce8-33e5-4197-9881-f89bb40636b5\") " Oct 30 17:22:29.673932 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:29.673906 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-openshift-ca" pod "f6657ce8-33e5-4197-9881-f89bb40636b5" (UID: "f6657ce8-33e5-4197-9881-f89bb40636b5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:29.675765 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:29.675743 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-xks-ca" pod "f6657ce8-33e5-4197-9881-f89bb40636b5" (UID: "f6657ce8-33e5-4197-9881-f89bb40636b5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:29.675954 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:29.675935 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6657ce8-33e5-4197-9881-f89bb40636b5-kube-api-access-wh7gq" pod "f6657ce8-33e5-4197-9881-f89bb40636b5" (UID: "f6657ce8-33e5-4197-9881-f89bb40636b5"). InnerVolumeSpecName "kube-api-access-wh7gq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:29.775252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:29.775219 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wh7gq\" (UniqueName: \"kubernetes.io/projected/f6657ce8-33e5-4197-9881-f89bb40636b5-kube-api-access-wh7gq\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:22:29.775252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:29.775246 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:22:29.775252 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:29.775255 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f6657ce8-33e5-4197-9881-f89bb40636b5-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:22:30.417185 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:30.417145 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" event={"ID":"f6657ce8-33e5-4197-9881-f89bb40636b5","Type":"ContainerDied","Data":"6d7f7b8dcb6c6519aa7a0792f74d6ff1fe2cb3b43f920792f874cefdc6dc327f"} Oct 30 17:22:30.417185 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:30.417175 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb" Oct 30 17:22:30.417415 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:30.417202 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d7f7b8dcb6c6519aa7a0792f74d6ff1fe2cb3b43f920792f874cefdc6dc327f" Oct 30 17:22:34.027057 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:34.027024 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt"] Oct 30 17:22:34.040232 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:34.040205 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-af8274de-dldxt"] Oct 30 17:22:34.965722 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:34.965687 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ce8ad726-6306-482d-9b5b-90c853b0d222" path="/var/lib/kubelet/pods/ce8ad726-6306-482d-9b5b-90c853b0d222/volumes" Oct 30 17:22:47.873515 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.873474 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg"] Oct 30 17:22:47.874090 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.874069 6400 cpu_manager.go:414] "containerMap: removing container" podUID="f6657ce8-33e5-4197-9881-f89bb40636b5" containerName="revoke-keys" Oct 30 17:22:47.874174 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.874093 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f6657ce8-33e5-4197-9881-f89bb40636b5" containerName="revoke-keys" Oct 30 17:22:47.874266 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.874253 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="f6657ce8-33e5-4197-9881-f89bb40636b5" containerName="revoke-keys" Oct 30 17:22:47.877636 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.877596 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:47.894539 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.894501 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:22:47.897094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.897064 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg"] Oct 30 17:22:47.907392 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.907360 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:22:47.942995 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.942957 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kbcg\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:47.942995 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.942998 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6d5jg\" (UniqueName: \"kubernetes.io/projected/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-kube-api-access-6d5jg\") pod \"maas-api-revoke-sub-0b7e8edb-7kbcg\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:47.943214 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:47.943112 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kbcg\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:48.043751 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:48.043711 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kbcg\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:48.043751 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:48.043752 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6d5jg\" (UniqueName: \"kubernetes.io/projected/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-kube-api-access-6d5jg\") pod \"maas-api-revoke-sub-0b7e8edb-7kbcg\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:48.044055 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:48.043806 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kbcg\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:48.044464 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:48.044436 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kbcg\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:48.046486 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:48.046461 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-7kbcg\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:48.068929 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:48.068897 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6d5jg\" (UniqueName: \"kubernetes.io/projected/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-kube-api-access-6d5jg\") pod \"maas-api-revoke-sub-0b7e8edb-7kbcg\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:48.190772 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:48.190683 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:48.495828 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:22:48.495801 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74ed1a5a_7687_4b48_929d_2fe3a8155ff4.slice/crio-fb1bc7633147649ae6ace989126e8534179e24eaaba1fb4b68d04f5de2f1ef3f WatchSource:0}: Error finding container fb1bc7633147649ae6ace989126e8534179e24eaaba1fb4b68d04f5de2f1ef3f: Status 404 returned error can't find the container with id fb1bc7633147649ae6ace989126e8534179e24eaaba1fb4b68d04f5de2f1ef3f Oct 30 17:22:48.496230 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:48.496205 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg"] Oct 30 17:22:48.523221 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:48.523187 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" event={"ID":"74ed1a5a-7687-4b48-929d-2fe3a8155ff4","Type":"ContainerStarted","Data":"fb1bc7633147649ae6ace989126e8534179e24eaaba1fb4b68d04f5de2f1ef3f"} Oct 30 17:22:49.530497 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:49.530457 6400 generic.go:356] "Generic (PLEG): container finished" podID="74ed1a5a-7687-4b48-929d-2fe3a8155ff4" containerID="272ef65a5f2b0f08e1547062553208c283ed11f9aa7e7255cff3c2f9f2869864" exitCode=0 Oct 30 17:22:49.530923 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:49.530551 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" event={"ID":"74ed1a5a-7687-4b48-929d-2fe3a8155ff4","Type":"ContainerDied","Data":"272ef65a5f2b0f08e1547062553208c283ed11f9aa7e7255cff3c2f9f2869864"} Oct 30 17:22:50.675120 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:50.675090 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:50.768804 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:50.768768 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-kube-api-access-6d5jg\" (UniqueName: \"kubernetes.io/projected/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-kube-api-access-6d5jg\") pod \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " Oct 30 17:22:50.768998 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:50.768861 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-openshift-ca\") pod \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " Oct 30 17:22:50.768998 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:50.768928 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-xks-ca\") pod \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\" (UID: \"74ed1a5a-7687-4b48-929d-2fe3a8155ff4\") " Oct 30 17:22:50.769226 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:50.769202 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-openshift-ca" pod "74ed1a5a-7687-4b48-929d-2fe3a8155ff4" (UID: "74ed1a5a-7687-4b48-929d-2fe3a8155ff4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:22:50.771157 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:50.771124 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-xks-ca" pod "74ed1a5a-7687-4b48-929d-2fe3a8155ff4" (UID: "74ed1a5a-7687-4b48-929d-2fe3a8155ff4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:22:50.771287 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:50.771193 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-kube-api-access-6d5jg" pod "74ed1a5a-7687-4b48-929d-2fe3a8155ff4" (UID: "74ed1a5a-7687-4b48-929d-2fe3a8155ff4"). InnerVolumeSpecName "kube-api-access-6d5jg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:22:50.869975 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:50.869893 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:22:50.869975 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:50.869923 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:22:50.869975 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:50.869932 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6d5jg\" (UniqueName: \"kubernetes.io/projected/74ed1a5a-7687-4b48-929d-2fe3a8155ff4-kube-api-access-6d5jg\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:22:51.542863 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:51.542836 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" Oct 30 17:22:51.542863 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:51.542842 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg" event={"ID":"74ed1a5a-7687-4b48-929d-2fe3a8155ff4","Type":"ContainerDied","Data":"fb1bc7633147649ae6ace989126e8534179e24eaaba1fb4b68d04f5de2f1ef3f"} Oct 30 17:22:51.543131 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:51.542885 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb1bc7633147649ae6ace989126e8534179e24eaaba1fb4b68d04f5de2f1ef3f" Oct 30 17:22:54.597239 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:22:54.597201 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:22:55.035218 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:55.035186 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms"] Oct 30 17:22:55.048405 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:55.048369 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3cd8bb6e-fjlms"] Oct 30 17:22:56.966372 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:56.966331 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256" path="/var/lib/kubelet/pods/fdc0a9eb-8c2c-40e0-a3d7-e4cdda97a256/volumes" Oct 30 17:22:58.647241 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.647209 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d"] Oct 30 17:22:58.647680 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.647625 6400 cpu_manager.go:414] "containerMap: removing container" podUID="74ed1a5a-7687-4b48-929d-2fe3a8155ff4" containerName="revoke-keys" Oct 30 17:22:58.647680 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.647637 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="74ed1a5a-7687-4b48-929d-2fe3a8155ff4" containerName="revoke-keys" Oct 30 17:22:58.647768 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.647709 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="74ed1a5a-7687-4b48-929d-2fe3a8155ff4" containerName="revoke-keys" Oct 30 17:22:58.651203 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.651186 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:58.672015 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.671985 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d"] Oct 30 17:22:58.681824 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.681793 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:22:58.681952 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.681832 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:22:58.744361 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.744331 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbcp\" (UniqueName: \"kubernetes.io/projected/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-kube-api-access-rvbcp\") pod \"maas-api-revoke-sub-3d4156f1-5kg9d\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:58.744548 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.744396 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5kg9d\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:58.744548 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.744504 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5kg9d\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:58.845282 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.845245 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5kg9d\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:58.845282 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.845297 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5kg9d\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:58.845562 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.845382 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbcp\" (UniqueName: \"kubernetes.io/projected/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-kube-api-access-rvbcp\") pod \"maas-api-revoke-sub-3d4156f1-5kg9d\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:58.845986 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.845961 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-5kg9d\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:58.847883 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.847861 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-5kg9d\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:58.870055 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.870029 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbcp\" (UniqueName: \"kubernetes.io/projected/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-kube-api-access-rvbcp\") pod \"maas-api-revoke-sub-3d4156f1-5kg9d\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:58.963802 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:58.963720 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:22:59.262520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:59.262490 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d"] Oct 30 17:22:59.262865 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:22:59.262832 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fd3187d_c8c3_441b_bf12_6e9d992cb4b2.slice/crio-298debb5db33e9552b3ce1cf727c917c7a99c5c1377bd52d034e7bedc045f61c WatchSource:0}: Error finding container 298debb5db33e9552b3ce1cf727c917c7a99c5c1377bd52d034e7bedc045f61c: Status 404 returned error can't find the container with id 298debb5db33e9552b3ce1cf727c917c7a99c5c1377bd52d034e7bedc045f61c Oct 30 17:22:59.586674 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:59.586627 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" event={"ID":"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2","Type":"ContainerStarted","Data":"bd59f7148ec18e38201c121d9489e571b24b90e3a558ee4474dad6455492b27e"} Oct 30 17:22:59.586884 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:59.586724 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" event={"ID":"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2","Type":"ContainerStarted","Data":"298debb5db33e9552b3ce1cf727c917c7a99c5c1377bd52d034e7bedc045f61c"} Oct 30 17:22:59.631816 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:22:59.631764 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" podStartSLOduration=1.631751857 podStartE2EDuration="1.631751857s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:22:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:22:59.629219537 +0000 UTC m=+3688.779500524" watchObservedRunningTime="2026-10-30 17:22:59.631751857 +0000 UTC m=+3688.782032845" Oct 30 17:23:00.593908 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:00.593870 6400 generic.go:356] "Generic (PLEG): container finished" podID="5fd3187d-c8c3-441b-bf12-6e9d992cb4b2" containerID="bd59f7148ec18e38201c121d9489e571b24b90e3a558ee4474dad6455492b27e" exitCode=0 Oct 30 17:23:00.594317 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:00.593961 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" event={"ID":"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2","Type":"ContainerDied","Data":"bd59f7148ec18e38201c121d9489e571b24b90e3a558ee4474dad6455492b27e"} Oct 30 17:23:01.736052 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:01.736023 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:23:01.876964 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:01.876880 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-xks-ca\") pod \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " Oct 30 17:23:01.876964 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:01.876927 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-openshift-ca\") pod \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " Oct 30 17:23:01.877170 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:01.877043 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-kube-api-access-rvbcp\" (UniqueName: \"kubernetes.io/projected/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-kube-api-access-rvbcp\") pod \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\" (UID: \"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2\") " Oct 30 17:23:01.877283 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:01.877262 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-openshift-ca" pod "5fd3187d-c8c3-441b-bf12-6e9d992cb4b2" (UID: "5fd3187d-c8c3-441b-bf12-6e9d992cb4b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:01.879216 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:01.879192 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-xks-ca" pod "5fd3187d-c8c3-441b-bf12-6e9d992cb4b2" (UID: "5fd3187d-c8c3-441b-bf12-6e9d992cb4b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:01.879351 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:01.879218 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-kube-api-access-rvbcp" pod "5fd3187d-c8c3-441b-bf12-6e9d992cb4b2" (UID: "5fd3187d-c8c3-441b-bf12-6e9d992cb4b2"). InnerVolumeSpecName "kube-api-access-rvbcp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:01.978580 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:01.978545 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvbcp\" (UniqueName: \"kubernetes.io/projected/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-kube-api-access-rvbcp\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:23:01.978580 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:01.978574 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:23:01.978580 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:01.978584 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:23:02.606235 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:02.606195 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" event={"ID":"5fd3187d-c8c3-441b-bf12-6e9d992cb4b2","Type":"ContainerDied","Data":"298debb5db33e9552b3ce1cf727c917c7a99c5c1377bd52d034e7bedc045f61c"} Oct 30 17:23:02.606422 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:02.606252 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="298debb5db33e9552b3ce1cf727c917c7a99c5c1377bd52d034e7bedc045f61c" Oct 30 17:23:02.606422 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:02.606213 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d" Oct 30 17:23:24.597206 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:23:24.597167 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:23:26.874866 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.874834 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d"] Oct 30 17:23:26.889420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.889390 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-5kg9d"] Oct 30 17:23:26.919676 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.919592 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5"] Oct 30 17:23:26.920099 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.920079 6400 cpu_manager.go:414] "containerMap: removing container" podUID="5fd3187d-c8c3-441b-bf12-6e9d992cb4b2" containerName="revoke-keys" Oct 30 17:23:26.920217 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.920103 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fd3187d-c8c3-441b-bf12-6e9d992cb4b2" containerName="revoke-keys" Oct 30 17:23:26.920285 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.920266 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fd3187d-c8c3-441b-bf12-6e9d992cb4b2" containerName="revoke-keys" Oct 30 17:23:26.923591 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.923568 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:26.941822 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.941798 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:23:26.941975 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.941873 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:23:26.945306 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.945273 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5"] Oct 30 17:23:26.967485 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:26.967454 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fd3187d-c8c3-441b-bf12-6e9d992cb4b2" path="/var/lib/kubelet/pods/5fd3187d-c8c3-441b-bf12-6e9d992cb4b2/volumes" Oct 30 17:23:27.031849 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.031807 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966a3405-8373-498d-8a87-557f1ec43192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zc6c5\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:27.031849 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.031856 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bbgwf\" (UniqueName: \"kubernetes.io/projected/966a3405-8373-498d-8a87-557f1ec43192-kube-api-access-bbgwf\") pod \"maas-api-revoke-sub-3d4156f1-zc6c5\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:27.032081 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.031971 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966a3405-8373-498d-8a87-557f1ec43192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zc6c5\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:27.133178 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.133094 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966a3405-8373-498d-8a87-557f1ec43192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zc6c5\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:27.133178 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.133137 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bbgwf\" (UniqueName: \"kubernetes.io/projected/966a3405-8373-498d-8a87-557f1ec43192-kube-api-access-bbgwf\") pod \"maas-api-revoke-sub-3d4156f1-zc6c5\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:27.133379 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.133224 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966a3405-8373-498d-8a87-557f1ec43192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zc6c5\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:27.133922 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.133899 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966a3405-8373-498d-8a87-557f1ec43192-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-zc6c5\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:27.135802 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.135778 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966a3405-8373-498d-8a87-557f1ec43192-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-zc6c5\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:27.158582 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.158551 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bbgwf\" (UniqueName: \"kubernetes.io/projected/966a3405-8373-498d-8a87-557f1ec43192-kube-api-access-bbgwf\") pod \"maas-api-revoke-sub-3d4156f1-zc6c5\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:27.236728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.236695 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:27.536067 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.536040 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5"] Oct 30 17:23:27.536778 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:23:27.536744 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod966a3405_8373_498d_8a87_557f1ec43192.slice/crio-34bf7b2ad7d7b0af7ba11c5ca9830e5ebae6fb976e0b6b1a3457e3063f93fa65 WatchSource:0}: Error finding container 34bf7b2ad7d7b0af7ba11c5ca9830e5ebae6fb976e0b6b1a3457e3063f93fa65: Status 404 returned error can't find the container with id 34bf7b2ad7d7b0af7ba11c5ca9830e5ebae6fb976e0b6b1a3457e3063f93fa65 Oct 30 17:23:27.783450 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:27.783406 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" event={"ID":"966a3405-8373-498d-8a87-557f1ec43192","Type":"ContainerStarted","Data":"34bf7b2ad7d7b0af7ba11c5ca9830e5ebae6fb976e0b6b1a3457e3063f93fa65"} Oct 30 17:23:28.790756 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:28.790718 6400 generic.go:356] "Generic (PLEG): container finished" podID="966a3405-8373-498d-8a87-557f1ec43192" containerID="c548cb5077034e7ba51aeecf379c0f5b07ca36859ba9dc8375844d8011811b38" exitCode=0 Oct 30 17:23:28.791165 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:28.790769 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" event={"ID":"966a3405-8373-498d-8a87-557f1ec43192","Type":"ContainerDied","Data":"c548cb5077034e7ba51aeecf379c0f5b07ca36859ba9dc8375844d8011811b38"} Oct 30 17:23:29.934208 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:29.934180 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:30.061843 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.061750 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/966a3405-8373-498d-8a87-557f1ec43192-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966a3405-8373-498d-8a87-557f1ec43192-maas-api-xks-ca\") pod \"966a3405-8373-498d-8a87-557f1ec43192\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " Oct 30 17:23:30.062008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.061867 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/966a3405-8373-498d-8a87-557f1ec43192-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966a3405-8373-498d-8a87-557f1ec43192-maas-api-openshift-ca\") pod \"966a3405-8373-498d-8a87-557f1ec43192\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " Oct 30 17:23:30.062008 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.061905 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/966a3405-8373-498d-8a87-557f1ec43192-kube-api-access-bbgwf\" (UniqueName: \"kubernetes.io/projected/966a3405-8373-498d-8a87-557f1ec43192-kube-api-access-bbgwf\") pod \"966a3405-8373-498d-8a87-557f1ec43192\" (UID: \"966a3405-8373-498d-8a87-557f1ec43192\") " Oct 30 17:23:30.062261 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.062237 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/966a3405-8373-498d-8a87-557f1ec43192-maas-api-openshift-ca" pod "966a3405-8373-498d-8a87-557f1ec43192" (UID: "966a3405-8373-498d-8a87-557f1ec43192"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:30.064189 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.064158 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/966a3405-8373-498d-8a87-557f1ec43192-kube-api-access-bbgwf" pod "966a3405-8373-498d-8a87-557f1ec43192" (UID: "966a3405-8373-498d-8a87-557f1ec43192"). InnerVolumeSpecName "kube-api-access-bbgwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:30.064288 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.064225 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/966a3405-8373-498d-8a87-557f1ec43192-maas-api-xks-ca" pod "966a3405-8373-498d-8a87-557f1ec43192" (UID: "966a3405-8373-498d-8a87-557f1ec43192"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:30.163204 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.163169 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/966a3405-8373-498d-8a87-557f1ec43192-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:23:30.163204 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.163204 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/966a3405-8373-498d-8a87-557f1ec43192-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:23:30.163441 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.163218 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bbgwf\" (UniqueName: \"kubernetes.io/projected/966a3405-8373-498d-8a87-557f1ec43192-kube-api-access-bbgwf\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:23:30.803894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.803851 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" event={"ID":"966a3405-8373-498d-8a87-557f1ec43192","Type":"ContainerDied","Data":"34bf7b2ad7d7b0af7ba11c5ca9830e5ebae6fb976e0b6b1a3457e3063f93fa65"} Oct 30 17:23:30.803894 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.803900 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="34bf7b2ad7d7b0af7ba11c5ca9830e5ebae6fb976e0b6b1a3457e3063f93fa65" Oct 30 17:23:30.804118 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:30.803868 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5" Oct 30 17:23:32.766670 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:32.766638 6400 scope.go:122] "RemoveContainer" containerID="3f05d77819d5c758d27ecd2ca53de71b89ca4b702f68d3de077be5a3df711562" Oct 30 17:23:32.781517 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:32.781495 6400 scope.go:122] "RemoveContainer" containerID="c0e0560c3f2c0f37781f4a0291bc4a0bd328643abc094b90f23653500c699051" Oct 30 17:23:54.472263 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.472227 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5"] Oct 30 17:23:54.485630 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.485590 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-zc6c5"] Oct 30 17:23:54.518645 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.518615 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4"] Oct 30 17:23:54.519070 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.519052 6400 cpu_manager.go:414] "containerMap: removing container" podUID="966a3405-8373-498d-8a87-557f1ec43192" containerName="revoke-keys" Oct 30 17:23:54.519070 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.519071 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="966a3405-8373-498d-8a87-557f1ec43192" containerName="revoke-keys" Oct 30 17:23:54.519222 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.519179 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="966a3405-8373-498d-8a87-557f1ec43192" containerName="revoke-keys" Oct 30 17:23:54.522307 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.522291 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.539798 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.539773 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:23:54.539943 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.539863 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:23:54.542790 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.542768 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4"] Oct 30 17:23:54.594901 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.594856 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-njsn4\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.595092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.594997 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flj8c\" (UniqueName: \"kubernetes.io/projected/87148bc5-22ed-41f6-99b9-98f5cd8be99e-kube-api-access-flj8c\") pod \"maas-api-revoke-sub-3d4156f1-njsn4\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.595092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.595073 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-njsn4\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.598336 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:23:54.598307 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:23:54.696559 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.696525 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flj8c\" (UniqueName: \"kubernetes.io/projected/87148bc5-22ed-41f6-99b9-98f5cd8be99e-kube-api-access-flj8c\") pod \"maas-api-revoke-sub-3d4156f1-njsn4\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.696747 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.696590 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-njsn4\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.696747 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.696692 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-njsn4\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.697432 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.697403 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-njsn4\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.699255 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.699233 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-njsn4\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.720963 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.720934 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flj8c\" (UniqueName: \"kubernetes.io/projected/87148bc5-22ed-41f6-99b9-98f5cd8be99e-kube-api-access-flj8c\") pod \"maas-api-revoke-sub-3d4156f1-njsn4\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.836042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.836008 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:54.966291 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:54.966254 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="966a3405-8373-498d-8a87-557f1ec43192" path="/var/lib/kubelet/pods/966a3405-8373-498d-8a87-557f1ec43192/volumes" Oct 30 17:23:55.028943 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:55.028913 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w"] Oct 30 17:23:55.032092 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:55.032060 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-hgt6w"] Oct 30 17:23:55.137981 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:55.137957 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4"] Oct 30 17:23:55.138534 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:23:55.138506 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87148bc5_22ed_41f6_99b9_98f5cd8be99e.slice/crio-713ac4498745306c36e7ce9c7bff7df30c2b5f4edd70ea218c58af1e6fb5792e WatchSource:0}: Error finding container 713ac4498745306c36e7ce9c7bff7df30c2b5f4edd70ea218c58af1e6fb5792e: Status 404 returned error can't find the container with id 713ac4498745306c36e7ce9c7bff7df30c2b5f4edd70ea218c58af1e6fb5792e Oct 30 17:23:55.945716 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:55.945677 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" event={"ID":"87148bc5-22ed-41f6-99b9-98f5cd8be99e","Type":"ContainerStarted","Data":"4df9d7e29d7f32e763e73f0f60b33cd09598145f41512c3ff2072164c85a788b"} Oct 30 17:23:55.946196 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:55.945789 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" event={"ID":"87148bc5-22ed-41f6-99b9-98f5cd8be99e","Type":"ContainerStarted","Data":"713ac4498745306c36e7ce9c7bff7df30c2b5f4edd70ea218c58af1e6fb5792e"} Oct 30 17:23:55.990414 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:55.990363 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" podStartSLOduration=1.990350284 podStartE2EDuration="1.990350284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:23:55.988513987 +0000 UTC m=+3745.138794976" watchObservedRunningTime="2026-10-30 17:23:55.990350284 +0000 UTC m=+3745.140631287" Oct 30 17:23:56.953513 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:56.953478 6400 generic.go:356] "Generic (PLEG): container finished" podID="87148bc5-22ed-41f6-99b9-98f5cd8be99e" containerID="4df9d7e29d7f32e763e73f0f60b33cd09598145f41512c3ff2072164c85a788b" exitCode=0 Oct 30 17:23:56.953961 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:56.953562 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" event={"ID":"87148bc5-22ed-41f6-99b9-98f5cd8be99e","Type":"ContainerDied","Data":"4df9d7e29d7f32e763e73f0f60b33cd09598145f41512c3ff2072164c85a788b"} Oct 30 17:23:56.969816 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:56.967332 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c" path="/var/lib/kubelet/pods/8bd1ef2f-c1af-4aee-95e0-f8d54a177f5c/volumes" Oct 30 17:23:58.093993 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.093969 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:23:58.231763 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.231664 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/87148bc5-22ed-41f6-99b9-98f5cd8be99e-kube-api-access-flj8c\" (UniqueName: \"kubernetes.io/projected/87148bc5-22ed-41f6-99b9-98f5cd8be99e-kube-api-access-flj8c\") pod \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " Oct 30 17:23:58.231950 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.231781 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-openshift-ca\") pod \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " Oct 30 17:23:58.231950 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.231803 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-xks-ca\") pod \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\" (UID: \"87148bc5-22ed-41f6-99b9-98f5cd8be99e\") " Oct 30 17:23:58.232199 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.232175 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-openshift-ca" pod "87148bc5-22ed-41f6-99b9-98f5cd8be99e" (UID: "87148bc5-22ed-41f6-99b9-98f5cd8be99e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:23:58.234079 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.234056 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-xks-ca" pod "87148bc5-22ed-41f6-99b9-98f5cd8be99e" (UID: "87148bc5-22ed-41f6-99b9-98f5cd8be99e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:23:58.234184 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.234167 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/87148bc5-22ed-41f6-99b9-98f5cd8be99e-kube-api-access-flj8c" pod "87148bc5-22ed-41f6-99b9-98f5cd8be99e" (UID: "87148bc5-22ed-41f6-99b9-98f5cd8be99e"). InnerVolumeSpecName "kube-api-access-flj8c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:23:58.332918 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.332881 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:23:58.332918 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.332912 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/87148bc5-22ed-41f6-99b9-98f5cd8be99e-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:23:58.332918 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.332922 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flj8c\" (UniqueName: \"kubernetes.io/projected/87148bc5-22ed-41f6-99b9-98f5cd8be99e-kube-api-access-flj8c\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:23:58.968569 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.968526 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" event={"ID":"87148bc5-22ed-41f6-99b9-98f5cd8be99e","Type":"ContainerDied","Data":"713ac4498745306c36e7ce9c7bff7df30c2b5f4edd70ea218c58af1e6fb5792e"} Oct 30 17:23:58.968884 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.968587 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="713ac4498745306c36e7ce9c7bff7df30c2b5f4edd70ea218c58af1e6fb5792e" Oct 30 17:23:58.968884 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:23:58.968537 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4" Oct 30 17:24:24.597563 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:24:24.597519 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:24:32.808099 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:32.808063 6400 scope.go:122] "RemoveContainer" containerID="bdc4ca6d98038ccf2c172e6444d6031e359a921059e78ced6f64c398dae0e020" Oct 30 17:24:32.817779 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:32.817761 6400 scope.go:122] "RemoveContainer" containerID="0054b8e5d56fc608f272848fd48ed66dd58de9273a79357f99becc7167461248" Oct 30 17:24:52.917376 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.917343 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68"] Oct 30 17:24:52.917895 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.917825 6400 cpu_manager.go:414] "containerMap: removing container" podUID="87148bc5-22ed-41f6-99b9-98f5cd8be99e" containerName="revoke-keys" Oct 30 17:24:52.917895 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.917841 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="87148bc5-22ed-41f6-99b9-98f5cd8be99e" containerName="revoke-keys" Oct 30 17:24:52.917975 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.917924 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="87148bc5-22ed-41f6-99b9-98f5cd8be99e" containerName="revoke-keys" Oct 30 17:24:52.920992 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.920976 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:52.942784 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.942756 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68"] Oct 30 17:24:52.953454 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.953423 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:24:52.953692 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.953638 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:24:52.976112 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.976079 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwfjb\" (UniqueName: \"kubernetes.io/projected/a53a3560-84ec-4f54-94a1-627f13885563-kube-api-access-kwfjb\") pod \"maas-api-revoke-sub-77dec780-wzm68\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:52.976431 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.976413 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dec780-wzm68\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:52.976558 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:52.976536 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dec780-wzm68\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:53.077862 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:53.077824 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwfjb\" (UniqueName: \"kubernetes.io/projected/a53a3560-84ec-4f54-94a1-627f13885563-kube-api-access-kwfjb\") pod \"maas-api-revoke-sub-77dec780-wzm68\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:53.078063 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:53.077899 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dec780-wzm68\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:53.078063 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:53.077938 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dec780-wzm68\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:53.078749 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:53.078727 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-77dec780-wzm68\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:53.080616 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:53.080585 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-xks-ca\") pod \"maas-api-revoke-sub-77dec780-wzm68\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:53.102564 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:53.102532 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwfjb\" (UniqueName: \"kubernetes.io/projected/a53a3560-84ec-4f54-94a1-627f13885563-kube-api-access-kwfjb\") pod \"maas-api-revoke-sub-77dec780-wzm68\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:53.233637 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:53.233590 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:53.533537 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:53.533509 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68"] Oct 30 17:24:53.533707 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:24:53.533659 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda53a3560_84ec_4f54_94a1_627f13885563.slice/crio-da7bb31ae088e3c9fe373b500a863857168f88b92a7e2a5e8a9073cdb253683d WatchSource:0}: Error finding container da7bb31ae088e3c9fe373b500a863857168f88b92a7e2a5e8a9073cdb253683d: Status 404 returned error can't find the container with id da7bb31ae088e3c9fe373b500a863857168f88b92a7e2a5e8a9073cdb253683d Oct 30 17:24:54.274985 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:54.274952 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" event={"ID":"a53a3560-84ec-4f54-94a1-627f13885563","Type":"ContainerStarted","Data":"af7cb0d750ed613db02f4419538ff1c45befa1f0397301251b8d28432114f2e1"} Oct 30 17:24:54.275409 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:54.275011 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" event={"ID":"a53a3560-84ec-4f54-94a1-627f13885563","Type":"ContainerStarted","Data":"da7bb31ae088e3c9fe373b500a863857168f88b92a7e2a5e8a9073cdb253683d"} Oct 30 17:24:54.318315 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:54.318266 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" podStartSLOduration=2.318252488 podStartE2EDuration="2.318252488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:24:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:24:54.317460111 +0000 UTC m=+3803.467741098" watchObservedRunningTime="2026-10-30 17:24:54.318252488 +0000 UTC m=+3803.468533476" Oct 30 17:24:54.597328 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:24:54.597242 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:24:55.281839 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:55.281797 6400 generic.go:356] "Generic (PLEG): container finished" podID="a53a3560-84ec-4f54-94a1-627f13885563" containerID="af7cb0d750ed613db02f4419538ff1c45befa1f0397301251b8d28432114f2e1" exitCode=0 Oct 30 17:24:55.282291 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:55.281875 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" event={"ID":"a53a3560-84ec-4f54-94a1-627f13885563","Type":"ContainerDied","Data":"af7cb0d750ed613db02f4419538ff1c45befa1f0397301251b8d28432114f2e1"} Oct 30 17:24:56.427484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:56.427458 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:56.508276 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:56.508236 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-xks-ca\") pod \"a53a3560-84ec-4f54-94a1-627f13885563\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " Oct 30 17:24:56.508482 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:56.508313 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a53a3560-84ec-4f54-94a1-627f13885563-kube-api-access-kwfjb\" (UniqueName: \"kubernetes.io/projected/a53a3560-84ec-4f54-94a1-627f13885563-kube-api-access-kwfjb\") pod \"a53a3560-84ec-4f54-94a1-627f13885563\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " Oct 30 17:24:56.508482 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:56.508353 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-openshift-ca\") pod \"a53a3560-84ec-4f54-94a1-627f13885563\" (UID: \"a53a3560-84ec-4f54-94a1-627f13885563\") " Oct 30 17:24:56.508821 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:56.508786 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-openshift-ca" pod "a53a3560-84ec-4f54-94a1-627f13885563" (UID: "a53a3560-84ec-4f54-94a1-627f13885563"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:24:56.511083 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:56.511050 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-xks-ca" pod "a53a3560-84ec-4f54-94a1-627f13885563" (UID: "a53a3560-84ec-4f54-94a1-627f13885563"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:24:56.511182 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:56.511148 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a53a3560-84ec-4f54-94a1-627f13885563-kube-api-access-kwfjb" pod "a53a3560-84ec-4f54-94a1-627f13885563" (UID: "a53a3560-84ec-4f54-94a1-627f13885563"). InnerVolumeSpecName "kube-api-access-kwfjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:24:56.610071 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:56.609989 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:24:56.610071 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:56.610025 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwfjb\" (UniqueName: \"kubernetes.io/projected/a53a3560-84ec-4f54-94a1-627f13885563-kube-api-access-kwfjb\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:24:56.610071 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:56.610041 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a53a3560-84ec-4f54-94a1-627f13885563-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:24:57.294020 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:57.293980 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" event={"ID":"a53a3560-84ec-4f54-94a1-627f13885563","Type":"ContainerDied","Data":"da7bb31ae088e3c9fe373b500a863857168f88b92a7e2a5e8a9073cdb253683d"} Oct 30 17:24:57.294020 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:57.294019 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68" Oct 30 17:24:57.294251 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:24:57.294035 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="da7bb31ae088e3c9fe373b500a863857168f88b92a7e2a5e8a9073cdb253683d" Oct 30 17:25:04.206545 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.206508 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp"] Oct 30 17:25:04.207174 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.206949 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a53a3560-84ec-4f54-94a1-627f13885563" containerName="revoke-keys" Oct 30 17:25:04.207174 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.206962 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a53a3560-84ec-4f54-94a1-627f13885563" containerName="revoke-keys" Oct 30 17:25:04.207174 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.207054 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a53a3560-84ec-4f54-94a1-627f13885563" containerName="revoke-keys" Oct 30 17:25:04.210193 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.210170 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.240937 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.240898 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:25:04.241136 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.241115 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:25:04.243736 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.243713 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp"] Oct 30 17:25:04.286187 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.285530 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56qvd\" (UniqueName: \"kubernetes.io/projected/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-kube-api-access-56qvd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.286187 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.285582 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.286187 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.285686 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.386431 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.386389 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56qvd\" (UniqueName: \"kubernetes.io/projected/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-kube-api-access-56qvd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.386431 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.386431 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.386663 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.386466 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.387108 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.387080 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.388875 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.388847 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.411213 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.411187 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56qvd\" (UniqueName: \"kubernetes.io/projected/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-kube-api-access-56qvd\") pod \"maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.523327 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.523296 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:04.851418 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:04.851390 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp"] Oct 30 17:25:04.852828 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:25:04.852800 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaa8b2a2_49ce_4dd1_b730_bfc9cc8a9de7.slice/crio-e1065d8e81d94292840cca21bee7d51d1775219cc69642760b9f8665b6fcfbff WatchSource:0}: Error finding container e1065d8e81d94292840cca21bee7d51d1775219cc69642760b9f8665b6fcfbff: Status 404 returned error can't find the container with id e1065d8e81d94292840cca21bee7d51d1775219cc69642760b9f8665b6fcfbff Oct 30 17:25:05.340988 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:05.340951 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" event={"ID":"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7","Type":"ContainerStarted","Data":"b11dddcf03d38f9d09ee6c8df424f7e15ac89daf5d8dc5e33d361351845e9295"} Oct 30 17:25:05.341444 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:05.341010 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" event={"ID":"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7","Type":"ContainerStarted","Data":"e1065d8e81d94292840cca21bee7d51d1775219cc69642760b9f8665b6fcfbff"} Oct 30 17:25:05.389380 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:05.389268 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" podStartSLOduration=1.3892532850000001 podStartE2EDuration="1.389253285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:25:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:25:05.387646995 +0000 UTC m=+3814.537927995" watchObservedRunningTime="2026-10-30 17:25:05.389253285 +0000 UTC m=+3814.539534293" Oct 30 17:25:06.348405 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:06.348368 6400 generic.go:356] "Generic (PLEG): container finished" podID="caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7" containerID="b11dddcf03d38f9d09ee6c8df424f7e15ac89daf5d8dc5e33d361351845e9295" exitCode=0 Oct 30 17:25:06.348875 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:06.348454 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" event={"ID":"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7","Type":"ContainerDied","Data":"b11dddcf03d38f9d09ee6c8df424f7e15ac89daf5d8dc5e33d361351845e9295"} Oct 30 17:25:07.490055 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:07.490029 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:07.619001 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:07.618917 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-xks-ca\") pod \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " Oct 30 17:25:07.619001 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:07.618986 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-kube-api-access-56qvd\" (UniqueName: \"kubernetes.io/projected/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-kube-api-access-56qvd\") pod \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " Oct 30 17:25:07.619204 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:07.619072 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-openshift-ca\") pod \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\" (UID: \"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7\") " Oct 30 17:25:07.619446 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:07.619422 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-openshift-ca" pod "caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7" (UID: "caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:07.621225 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:07.621200 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-xks-ca" pod "caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7" (UID: "caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:07.621329 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:07.621261 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-kube-api-access-56qvd" pod "caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7" (UID: "caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7"). InnerVolumeSpecName "kube-api-access-56qvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:07.720234 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:07.720199 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:25:07.720234 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:07.720230 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56qvd\" (UniqueName: \"kubernetes.io/projected/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-kube-api-access-56qvd\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:25:07.720234 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:07.720239 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:25:08.360723 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:08.360689 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" event={"ID":"caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7","Type":"ContainerDied","Data":"e1065d8e81d94292840cca21bee7d51d1775219cc69642760b9f8665b6fcfbff"} Oct 30 17:25:08.360723 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:08.360719 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp" Oct 30 17:25:08.360963 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:08.360738 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1065d8e81d94292840cca21bee7d51d1775219cc69642760b9f8665b6fcfbff" Oct 30 17:25:17.031494 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:17.031458 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95"] Oct 30 17:25:17.044274 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:17.044245 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-p2d95"] Oct 30 17:25:18.168241 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:18.168210 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp"] Oct 30 17:25:18.181375 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:18.181344 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-single-main-57f1de-dzhgp"] Oct 30 17:25:18.965926 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:18.965892 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4" path="/var/lib/kubelet/pods/c61e65ad-d8c3-4786-b0ba-9ef9f30f05b4/volumes" Oct 30 17:25:18.966307 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:18.966295 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7" path="/var/lib/kubelet/pods/caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7/volumes" Oct 30 17:25:24.598165 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:25:24.598126 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:25:28.029072 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:28.029039 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8"] Oct 30 17:25:28.041937 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:28.041911 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-k7tx8"] Oct 30 17:25:28.965372 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:28.965337 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e29d7aee-74d8-472f-8b41-87720fb09c09" path="/var/lib/kubelet/pods/e29d7aee-74d8-472f-8b41-87720fb09c09/volumes" Oct 30 17:25:32.853198 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:32.853159 6400 scope.go:122] "RemoveContainer" containerID="a72cba5f0ed38139f51e1a94f1d99fcd74470966834e053adc619cd12116d40f" Oct 30 17:25:37.686346 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.686313 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4"] Oct 30 17:25:37.699502 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.699469 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-njsn4"] Oct 30 17:25:37.750052 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.750019 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f"] Oct 30 17:25:37.750424 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.750412 6400 cpu_manager.go:414] "containerMap: removing container" podUID="caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7" containerName="revoke-keys" Oct 30 17:25:37.750474 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.750425 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7" containerName="revoke-keys" Oct 30 17:25:37.750522 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.750494 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="caa8b2a2-49ce-4dd1-b730-bfc9cc8a9de7" containerName="revoke-keys" Oct 30 17:25:37.753522 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.753505 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:37.770740 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.770710 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-qgmls\"" Oct 30 17:25:37.770906 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.770787 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 17:25:37.773506 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.773476 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f"] Oct 30 17:25:37.818237 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.818205 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581faf88-cae4-450f-af3c-234a11580a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmj2f\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:37.818457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.818254 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581faf88-cae4-450f-af3c-234a11580a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmj2f\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:37.818457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.818277 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gl6mh\" (UniqueName: \"kubernetes.io/projected/581faf88-cae4-450f-af3c-234a11580a57-kube-api-access-gl6mh\") pod \"maas-api-revoke-sub-3d4156f1-cmj2f\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:37.919277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.919243 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581faf88-cae4-450f-af3c-234a11580a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmj2f\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:37.919484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.919292 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581faf88-cae4-450f-af3c-234a11580a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmj2f\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:37.919484 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.919318 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gl6mh\" (UniqueName: \"kubernetes.io/projected/581faf88-cae4-450f-af3c-234a11580a57-kube-api-access-gl6mh\") pod \"maas-api-revoke-sub-3d4156f1-cmj2f\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:37.919946 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.919922 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581faf88-cae4-450f-af3c-234a11580a57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmj2f\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:37.921872 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.921851 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581faf88-cae4-450f-af3c-234a11580a57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-cmj2f\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:37.943877 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:37.943815 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gl6mh\" (UniqueName: \"kubernetes.io/projected/581faf88-cae4-450f-af3c-234a11580a57-kube-api-access-gl6mh\") pod \"maas-api-revoke-sub-3d4156f1-cmj2f\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:38.066585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:38.066552 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:38.368322 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:38.368296 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f"] Oct 30 17:25:38.369334 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:25:38.369307 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod581faf88_cae4_450f_af3c_234a11580a57.slice/crio-551f9cf7216b0f1f9eb1c79273dfbd4a5ff8e61c4de7a4818ea350485d7ce8f0 WatchSource:0}: Error finding container 551f9cf7216b0f1f9eb1c79273dfbd4a5ff8e61c4de7a4818ea350485d7ce8f0: Status 404 returned error can't find the container with id 551f9cf7216b0f1f9eb1c79273dfbd4a5ff8e61c4de7a4818ea350485d7ce8f0 Oct 30 17:25:38.528136 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:38.528082 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" event={"ID":"581faf88-cae4-450f-af3c-234a11580a57","Type":"ContainerStarted","Data":"551f9cf7216b0f1f9eb1c79273dfbd4a5ff8e61c4de7a4818ea350485d7ce8f0"} Oct 30 17:25:38.965882 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:38.965801 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="87148bc5-22ed-41f6-99b9-98f5cd8be99e" path="/var/lib/kubelet/pods/87148bc5-22ed-41f6-99b9-98f5cd8be99e/volumes" Oct 30 17:25:39.534931 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:39.534897 6400 generic.go:356] "Generic (PLEG): container finished" podID="581faf88-cae4-450f-af3c-234a11580a57" containerID="a155316f76763b7e013f5f923b4a891c0052166383b3f47cf09d7368fcf30d5e" exitCode=0 Oct 30 17:25:39.535126 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:39.534987 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" event={"ID":"581faf88-cae4-450f-af3c-234a11580a57","Type":"ContainerDied","Data":"a155316f76763b7e013f5f923b4a891c0052166383b3f47cf09d7368fcf30d5e"} Oct 30 17:25:40.677389 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:40.677364 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:40.747368 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:40.747333 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/581faf88-cae4-450f-af3c-234a11580a57-kube-api-access-gl6mh\" (UniqueName: \"kubernetes.io/projected/581faf88-cae4-450f-af3c-234a11580a57-kube-api-access-gl6mh\") pod \"581faf88-cae4-450f-af3c-234a11580a57\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " Oct 30 17:25:40.747575 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:40.747380 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/581faf88-cae4-450f-af3c-234a11580a57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581faf88-cae4-450f-af3c-234a11580a57-maas-api-openshift-ca\") pod \"581faf88-cae4-450f-af3c-234a11580a57\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " Oct 30 17:25:40.747575 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:40.747472 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/581faf88-cae4-450f-af3c-234a11580a57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581faf88-cae4-450f-af3c-234a11580a57-maas-api-xks-ca\") pod \"581faf88-cae4-450f-af3c-234a11580a57\" (UID: \"581faf88-cae4-450f-af3c-234a11580a57\") " Oct 30 17:25:40.747828 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:40.747802 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/581faf88-cae4-450f-af3c-234a11580a57-maas-api-openshift-ca" pod "581faf88-cae4-450f-af3c-234a11580a57" (UID: "581faf88-cae4-450f-af3c-234a11580a57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:25:40.749665 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:40.749632 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/581faf88-cae4-450f-af3c-234a11580a57-maas-api-xks-ca" pod "581faf88-cae4-450f-af3c-234a11580a57" (UID: "581faf88-cae4-450f-af3c-234a11580a57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:25:40.749773 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:40.749753 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/581faf88-cae4-450f-af3c-234a11580a57-kube-api-access-gl6mh" pod "581faf88-cae4-450f-af3c-234a11580a57" (UID: "581faf88-cae4-450f-af3c-234a11580a57"). InnerVolumeSpecName "kube-api-access-gl6mh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:25:40.848519 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:40.848441 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gl6mh\" (UniqueName: \"kubernetes.io/projected/581faf88-cae4-450f-af3c-234a11580a57-kube-api-access-gl6mh\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.848519 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:40.848470 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/581faf88-cae4-450f-af3c-234a11580a57-maas-api-openshift-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:25:40.848519 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:40.848480 6400 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/581faf88-cae4-450f-af3c-234a11580a57-maas-api-xks-ca\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:25:41.548247 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:41.548216 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" event={"ID":"581faf88-cae4-450f-af3c-234a11580a57","Type":"ContainerDied","Data":"551f9cf7216b0f1f9eb1c79273dfbd4a5ff8e61c4de7a4818ea350485d7ce8f0"} Oct 30 17:25:41.548431 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:41.548264 6400 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="551f9cf7216b0f1f9eb1c79273dfbd4a5ff8e61c4de7a4818ea350485d7ce8f0" Oct 30 17:25:41.548431 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:25:41.548230 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f" Oct 30 17:25:54.597262 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:25:54.597225 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:26:16.028467 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:16.028430 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd"] Oct 30 17:26:16.041644 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:16.041615 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-nrrvd"] Oct 30 17:26:16.965555 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:16.965522 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="677cc0e1-27a8-48b7-9986-913d69d2bb10" path="/var/lib/kubelet/pods/677cc0e1-27a8-48b7-9986-913d69d2bb10/volumes" Oct 30 17:26:24.597531 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:26:24.597491 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:26:27.028748 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:27.028711 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr"] Oct 30 17:26:27.041861 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:27.041828 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-788wr"] Oct 30 17:26:28.966050 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:28.966016 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d568734f-189b-4a9b-976f-efdb2fdd844c" path="/var/lib/kubelet/pods/d568734f-189b-4a9b-976f-efdb2fdd844c/volumes" Oct 30 17:26:32.876180 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:32.876149 6400 scope.go:122] "RemoveContainer" containerID="f3635933b5c451f10860da921ad093b65c8ac23f722ede7884aeeaa422a97476" Oct 30 17:26:32.885675 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:32.885652 6400 scope.go:122] "RemoveContainer" containerID="96fb9fd2e0cecb5c66863cc2f77f1ea369a4e9b385a093370786609f2083b027" Oct 30 17:26:49.029929 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:49.029884 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr"] Oct 30 17:26:49.043297 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:49.043254 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-8tssr"] Oct 30 17:26:50.966772 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:26:50.966740 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee36df14-bf62-4539-b242-0993c87d7011" path="/var/lib/kubelet/pods/ee36df14-bf62-4539-b242-0993c87d7011/volumes" Oct 30 17:26:54.597659 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:26:54.597620 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:27:10.033026 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:10.032993 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp"] Oct 30 17:27:10.045135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:10.045106 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-tr5hp"] Oct 30 17:27:10.966418 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:10.966381 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca133507-3790-404c-bb3d-1e631b7048dc" path="/var/lib/kubelet/pods/ca133507-3790-404c-bb3d-1e631b7048dc/volumes" Oct 30 17:27:24.599242 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:27:24.599200 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:27:30.029746 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:30.029709 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb"] Oct 30 17:27:30.042671 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:30.042637 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-mwmrb"] Oct 30 17:27:30.966546 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:30.966510 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f6657ce8-33e5-4197-9881-f89bb40636b5" path="/var/lib/kubelet/pods/f6657ce8-33e5-4197-9881-f89bb40636b5/volumes" Oct 30 17:27:32.913215 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:32.913182 6400 scope.go:122] "RemoveContainer" containerID="83efc4c5fa17481b3f44cd34d922484ee3011f775be152fcab5744589097047b" Oct 30 17:27:32.922493 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:32.922470 6400 scope.go:122] "RemoveContainer" containerID="06bb68969718694b27f24c82104e9fcbad323e98422d63a2923c4695f236d0e6" Oct 30 17:27:51.034189 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:51.034154 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg"] Oct 30 17:27:51.047325 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:51.047297 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-7kbcg"] Oct 30 17:27:52.982186 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:27:52.982148 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="74ed1a5a-7687-4b48-929d-2fe3a8155ff4" path="/var/lib/kubelet/pods/74ed1a5a-7687-4b48-929d-2fe3a8155ff4/volumes" Oct 30 17:27:54.596772 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:27:54.596737 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:28:24.595843 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:28:24.595807 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:28:32.948712 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:32.948686 6400 scope.go:122] "RemoveContainer" containerID="5aac1a1324a492d4e4d883b563a81ce2d87771864ebdd5e1e700d9f2938d89a0" Oct 30 17:28:32.957654 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:32.957632 6400 scope.go:122] "RemoveContainer" containerID="e6f90858427133ce644bf0fc79842d38174998c8725c22c753ab2086bc486ae9" Oct 30 17:28:32.966645 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:32.966621 6400 scope.go:122] "RemoveContainer" containerID="544edae1c1b3936497846995e63175361357b867e177b23f617651ba239dd0de" Oct 30 17:28:47.577181 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.577149 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8"] Oct 30 17:28:47.577906 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.577887 6400 cpu_manager.go:414] "containerMap: removing container" podUID="581faf88-cae4-450f-af3c-234a11580a57" containerName="revoke-keys" Oct 30 17:28:47.577962 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.577911 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="581faf88-cae4-450f-af3c-234a11580a57" containerName="revoke-keys" Oct 30 17:28:47.578025 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.578015 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="581faf88-cae4-450f-af3c-234a11580a57" containerName="revoke-keys" Oct 30 17:28:47.581228 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.581208 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.600422 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.600394 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8"] Oct 30 17:28:47.611512 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.611490 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e2e8d2\"/\"kube-root-ca.crt\"" Oct 30 17:28:47.611766 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.611747 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-e2e8d2\"/\"openshift-service-ca.crt\"" Oct 30 17:28:47.611840 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.611769 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e2e8d2\"/\"default-dockercfg-x7kj8\"" Oct 30 17:28:47.611886 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.611865 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-e2e8d2\"/\"simulator-single-main-e2e8d2-kserve-self-signed-certs\"" Oct 30 17:28:47.705586 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.705554 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21cfe5ff-f1aa-4463-b263-c59f7153825b-tls-certs\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.705794 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.705690 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-dshm\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.705794 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.705751 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-tmp-dir\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.705794 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.705784 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-home\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.705920 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.705832 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cblhw\" (UniqueName: \"kubernetes.io/projected/21cfe5ff-f1aa-4463-b263-c59f7153825b-kube-api-access-cblhw\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.705973 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.705927 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-model-cache\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.807075 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.807035 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-dshm\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.807273 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.807098 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-tmp-dir\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.807273 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.807134 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-home\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.807273 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.807158 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cblhw\" (UniqueName: \"kubernetes.io/projected/21cfe5ff-f1aa-4463-b263-c59f7153825b-kube-api-access-cblhw\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.807273 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.807204 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-model-cache\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.807273 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.807262 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21cfe5ff-f1aa-4463-b263-c59f7153825b-tls-certs\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.807924 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.807895 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-home\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.808118 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.807983 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-tmp-dir\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.808118 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.808035 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-model-cache\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.810747 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.810723 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21cfe5ff-f1aa-4463-b263-c59f7153825b-tls-certs\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.817462 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.817430 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-dshm\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.833151 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.833096 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cblhw\" (UniqueName: \"kubernetes.io/projected/21cfe5ff-f1aa-4463-b263-c59f7153825b-kube-api-access-cblhw\") pod \"simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:47.893247 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:47.893214 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:48.192614 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:48.192576 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8"] Oct 30 17:28:48.193915 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:28:48.193888 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21cfe5ff_f1aa_4463_b263_c59f7153825b.slice/crio-112bc052feb6fedd7b559b8e5a9b1f276aa06b9b20b8364de66438611068e23e WatchSource:0}: Error finding container 112bc052feb6fedd7b559b8e5a9b1f276aa06b9b20b8364de66438611068e23e: Status 404 returned error can't find the container with id 112bc052feb6fedd7b559b8e5a9b1f276aa06b9b20b8364de66438611068e23e Oct 30 17:28:48.594595 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:48.594549 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" event={"ID":"21cfe5ff-f1aa-4463-b263-c59f7153825b","Type":"ContainerStarted","Data":"f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce"} Oct 30 17:28:48.594595 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:48.594620 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" event={"ID":"21cfe5ff-f1aa-4463-b263-c59f7153825b","Type":"ContainerStarted","Data":"112bc052feb6fedd7b559b8e5a9b1f276aa06b9b20b8364de66438611068e23e"} Oct 30 17:28:48.637863 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:48.637814 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" podStartSLOduration=1.637803165 podStartE2EDuration="1.637803165s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:28:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:28:48.637246514 +0000 UTC m=+4037.787527502" watchObservedRunningTime="2026-10-30 17:28:48.637803165 +0000 UTC m=+4037.788084197" Oct 30 17:28:48.893528 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:48.893460 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:54.597445 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:28:54.597408 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:28:57.893731 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:57.893688 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:57.906464 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:57.906436 6400 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:28:58.659872 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:28:58.659840 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:29:23.042698 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:23.042586 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8"] Oct 30 17:29:24.597851 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:29:24.597813 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:29:32.994531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:32.994496 6400 scope.go:122] "RemoveContainer" containerID="c548cb5077034e7ba51aeecf379c0f5b07ca36859ba9dc8375844d8011811b38" Oct 30 17:29:33.003531 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:33.003507 6400 scope.go:122] "RemoveContainer" containerID="bd59f7148ec18e38201c121d9489e571b24b90e3a558ee4474dad6455492b27e" Oct 30 17:29:33.012023 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:33.011999 6400 scope.go:122] "RemoveContainer" containerID="272ef65a5f2b0f08e1547062553208c283ed11f9aa7e7255cff3c2f9f2869864" Oct 30 17:29:38.064238 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.064173 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" podUID="21cfe5ff-f1aa-4463-b263-c59f7153825b" containerName="main" containerID="cri-o://f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce" gracePeriod=45 Oct 30 17:29:38.316260 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.316196 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:29:38.412730 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.412702 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-dshm\") pod \"21cfe5ff-f1aa-4463-b263-c59f7153825b\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " Oct 30 17:29:38.412914 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.412745 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/21cfe5ff-f1aa-4463-b263-c59f7153825b-tls-certs\" (UniqueName: \"kubernetes.io/secret/21cfe5ff-f1aa-4463-b263-c59f7153825b-tls-certs\") pod \"21cfe5ff-f1aa-4463-b263-c59f7153825b\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " Oct 30 17:29:38.412914 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.412766 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-home\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-home\") pod \"21cfe5ff-f1aa-4463-b263-c59f7153825b\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " Oct 30 17:29:38.412914 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.412795 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-model-cache\") pod \"21cfe5ff-f1aa-4463-b263-c59f7153825b\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " Oct 30 17:29:38.412914 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.412887 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21cfe5ff-f1aa-4463-b263-c59f7153825b-kube-api-access-cblhw\" (UniqueName: \"kubernetes.io/projected/21cfe5ff-f1aa-4463-b263-c59f7153825b-kube-api-access-cblhw\") pod \"21cfe5ff-f1aa-4463-b263-c59f7153825b\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " Oct 30 17:29:38.412914 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.412909 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-tmp-dir\") pod \"21cfe5ff-f1aa-4463-b263-c59f7153825b\" (UID: \"21cfe5ff-f1aa-4463-b263-c59f7153825b\") " Oct 30 17:29:38.413198 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.413162 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-model-cache" pod "21cfe5ff-f1aa-4463-b263-c59f7153825b" (UID: "21cfe5ff-f1aa-4463-b263-c59f7153825b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:38.413320 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.413292 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-home" pod "21cfe5ff-f1aa-4463-b263-c59f7153825b" (UID: "21cfe5ff-f1aa-4463-b263-c59f7153825b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:38.413411 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.413312 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-tmp-dir" pod "21cfe5ff-f1aa-4463-b263-c59f7153825b" (UID: "21cfe5ff-f1aa-4463-b263-c59f7153825b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:38.413411 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.413358 6400 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-model-cache\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:29:38.415135 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.415103 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-dshm" pod "21cfe5ff-f1aa-4463-b263-c59f7153825b" (UID: "21cfe5ff-f1aa-4463-b263-c59f7153825b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:29:38.415253 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.415168 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21cfe5ff-f1aa-4463-b263-c59f7153825b-kube-api-access-cblhw" pod "21cfe5ff-f1aa-4463-b263-c59f7153825b" (UID: "21cfe5ff-f1aa-4463-b263-c59f7153825b"). InnerVolumeSpecName "kube-api-access-cblhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:29:38.415395 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.415370 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21cfe5ff-f1aa-4463-b263-c59f7153825b-tls-certs" pod "21cfe5ff-f1aa-4463-b263-c59f7153825b" (UID: "21cfe5ff-f1aa-4463-b263-c59f7153825b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 17:29:38.514251 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.514212 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cblhw\" (UniqueName: \"kubernetes.io/projected/21cfe5ff-f1aa-4463-b263-c59f7153825b-kube-api-access-cblhw\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:29:38.514251 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.514246 6400 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-tmp-dir\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:29:38.514251 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.514258 6400 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-dshm\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:29:38.514516 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.514266 6400 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/21cfe5ff-f1aa-4463-b263-c59f7153825b-tls-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:29:38.514516 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.514274 6400 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/21cfe5ff-f1aa-4463-b263-c59f7153825b-home\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:29:38.865010 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.864956 6400 generic.go:356] "Generic (PLEG): container finished" podID="21cfe5ff-f1aa-4463-b263-c59f7153825b" containerID="f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce" exitCode=0 Oct 30 17:29:38.865206 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.865026 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" Oct 30 17:29:38.865206 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.865028 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" event={"ID":"21cfe5ff-f1aa-4463-b263-c59f7153825b","Type":"ContainerDied","Data":"f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce"} Oct 30 17:29:38.865206 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.865137 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8" event={"ID":"21cfe5ff-f1aa-4463-b263-c59f7153825b","Type":"ContainerDied","Data":"112bc052feb6fedd7b559b8e5a9b1f276aa06b9b20b8364de66438611068e23e"} Oct 30 17:29:38.865206 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.865156 6400 scope.go:122] "RemoveContainer" containerID="f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce" Oct 30 17:29:38.875393 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.875375 6400 scope.go:122] "RemoveContainer" containerID="f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce" Oct 30 17:29:38.875671 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:29:38.875650 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce\": container with ID starting with f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce not found: ID does not exist" containerID="f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce" Oct 30 17:29:38.875728 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.875682 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce"} err="failed to get container status \"f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce\": rpc error: code = NotFound desc = could not find container \"f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce\": container with ID starting with f1391f245f87167c4512afaa80b9e8af7de574609e996e8b07647502597373ce not found: ID does not exist" Oct 30 17:29:38.919329 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.919293 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8"] Oct 30 17:29:38.921020 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.921000 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-e2e8d2/simulator-single-main-e2e8d2-kserve-7cc9568dc8-t59t8"] Oct 30 17:29:38.966807 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:38.966766 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21cfe5ff-f1aa-4463-b263-c59f7153825b" path="/var/lib/kubelet/pods/21cfe5ff-f1aa-4463-b263-c59f7153825b/volumes" Oct 30 17:29:54.598796 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:29:54.598755 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:29:57.042463 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:57.042416 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68"] Oct 30 17:29:57.054867 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:57.054829 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-77dec780-wzm68"] Oct 30 17:29:58.985311 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:29:58.984580 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a53a3560-84ec-4f54-94a1-627f13885563" path="/var/lib/kubelet/pods/a53a3560-84ec-4f54-94a1-627f13885563/volumes" Oct 30 17:30:19.007340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:19.007292 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qghkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:19.007903 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:19.007380 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:19.646752 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:19.646711 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:19.647053 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:19.647022 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:24.597854 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:30:24.597809 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:30:33.047627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:33.046188 6400 scope.go:122] "RemoveContainer" containerID="4df9d7e29d7f32e763e73f0f60b33cd09598145f41512c3ff2072164c85a788b" Oct 30 17:30:34.054625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:34.053709 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-qghkn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:34.054625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:34.053788 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:34.650627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:34.646732 6400 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-4nbqj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 17:30:34.650627 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:34.646807 6400 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 17:30:41.028723 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:41.028640 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f"] Oct 30 17:30:41.041722 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:41.041689 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-cmj2f"] Oct 30 17:30:42.965483 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:30:42.965445 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="581faf88-cae4-450f-af3c-234a11580a57" path="/var/lib/kubelet/pods/581faf88-cae4-450f-af3c-234a11580a57/volumes" Oct 30 17:30:54.599233 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:30:54.599180 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:31:24.597672 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:31:24.597626 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:31:33.160626 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:33.160561 6400 scope.go:122] "RemoveContainer" containerID="af7cb0d750ed613db02f4419538ff1c45befa1f0397301251b8d28432114f2e1" Oct 30 17:31:33.169732 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:33.169699 6400 scope.go:122] "RemoveContainer" containerID="b11dddcf03d38f9d09ee6c8df424f7e15ac89daf5d8dc5e33d361351845e9295" Oct 30 17:31:39.706069 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.706035 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4wp2c"] Oct 30 17:31:39.706486 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.706469 6400 cpu_manager.go:414] "containerMap: removing container" podUID="21cfe5ff-f1aa-4463-b263-c59f7153825b" containerName="main" Oct 30 17:31:39.706538 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.706489 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21cfe5ff-f1aa-4463-b263-c59f7153825b" containerName="main" Oct 30 17:31:39.706717 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.706701 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="21cfe5ff-f1aa-4463-b263-c59f7153825b" containerName="main" Oct 30 17:31:39.709526 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.709509 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.735407 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.735381 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-metrics-certs\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.735554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.735414 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-openshift-service-ca\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.735554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.735436 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmkjt\" (UniqueName: \"kubernetes.io/projected/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-kube-api-access-rmkjt\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.735554 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.735480 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-maas-api-tls\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.741186 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.741162 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-5jjc6\"" Oct 30 17:31:39.741329 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.741243 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 17:31:39.741691 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.741668 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 17:31:39.759566 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.755627 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4wp2c"] Oct 30 17:31:39.835999 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.835965 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-metrics-certs\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.835999 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.836003 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-openshift-service-ca\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.836298 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.836030 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmkjt\" (UniqueName: \"kubernetes.io/projected/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-kube-api-access-rmkjt\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.836298 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.836152 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-maas-api-tls\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.836780 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.836756 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-openshift-service-ca\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.838827 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.838807 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-metrics-certs\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.838924 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.838866 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-maas-api-tls\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:39.862544 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:39.862513 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmkjt\" (UniqueName: \"kubernetes.io/projected/0c1d6a5b-d18b-40af-8199-2ddc02dae6dd-kube-api-access-rmkjt\") pod \"maas-api-595cc648c-4wp2c\" (UID: \"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd\") " pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:40.022261 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:40.022221 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:40.338423 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:40.338392 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-595cc648c-4wp2c"] Oct 30 17:31:40.339167 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:31:40.339141 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c1d6a5b_d18b_40af_8199_2ddc02dae6dd.slice/crio-fb06c69f921827181561fc1abbb77f9337de42cdbebdbaeb2a3c02902ae99cb8 WatchSource:0}: Error finding container fb06c69f921827181561fc1abbb77f9337de42cdbebdbaeb2a3c02902ae99cb8: Status 404 returned error can't find the container with id fb06c69f921827181561fc1abbb77f9337de42cdbebdbaeb2a3c02902ae99cb8 Oct 30 17:31:40.340503 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:40.340488 6400 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 17:31:40.579779 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:40.579741 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" event={"ID":"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd","Type":"ContainerStarted","Data":"fb06c69f921827181561fc1abbb77f9337de42cdbebdbaeb2a3c02902ae99cb8"} Oct 30 17:31:41.588011 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:41.587962 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" event={"ID":"0c1d6a5b-d18b-40af-8199-2ddc02dae6dd","Type":"ContainerStarted","Data":"a323dff918f37a0d634cc2c8c77c1513c933f8d899bba701ec0267d7377c5366"} Oct 30 17:31:41.588453 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:41.588159 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:41.631017 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:41.630939 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" podStartSLOduration=2.261795142 podStartE2EDuration="2.63092384s" totalImagesPullingTime="369.128698ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 17:31:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:31:41.628593707 +0000 UTC m=+4210.778874696" watchObservedRunningTime="2026-10-30 17:31:41.63092384 +0000 UTC m=+4210.781204828" Oct 30 17:31:47.599723 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:31:47.599690 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-595cc648c-4wp2c" Oct 30 17:31:54.596818 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:31:54.596777 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:32:24.598752 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:32:24.598655 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:32:33.196130 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:32:33.196095 6400 scope.go:122] "RemoveContainer" containerID="a155316f76763b7e013f5f923b4a891c0052166383b3f47cf09d7368fcf30d5e" Oct 30 17:32:54.597772 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:32:54.597736 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:33:24.598417 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:33:24.598371 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:33:26.976766 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:26.976722 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vg6k7_d953cc88-99a1-4389-aec1-f8cfda8e7de5/manager/0.log" Oct 30 17:33:27.424703 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:27.424669 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vt287_2802841b-377f-409c-ba62-eb90a400265a/manager/0.log" Oct 30 17:33:27.939458 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:27.939429 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zgr4b_aa757e1d-7dbd-47cf-bc6f-54dacda93285/server/0.log" Oct 30 17:33:28.143148 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:28.143113 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-5fmhl_7ac575cb-7c53-489b-b05c-09ca444bfabb/manager/0.log" Oct 30 17:33:28.369325 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:28.369287 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-k7579_02309873-b9e3-4457-85d2-75c94131b212/manager/0.log" Oct 30 17:33:29.077778 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:29.077746 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-xpdtn_5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22/copy-manifests/0.log" Oct 30 17:33:29.119897 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:29.119867 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-xpdtn_5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22/manager/0.log" Oct 30 17:33:30.137429 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:30.137399 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-4wp2c_0c1d6a5b-d18b-40af-8199-2ddc02dae6dd/maas-api/0.log" Oct 30 17:33:31.917585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:31.917549 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-rkmxm_c4b56d9a-9f54-417a-89d0-bde1880c0dda/postgres/0.log" Oct 30 17:33:34.122662 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:34.122632 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6xzjs_79795cb8-7a9c-4aa0-8bd1-1dec782af242/kuadrant-console-plugin/0.log" Oct 30 17:33:35.528838 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:35.528802 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-kz9k9_ffda5415-4ff3-4fce-abfc-c8e44f905c5b/istio-proxy/0.log" Oct 30 17:33:36.344748 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:36.344714 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-4nbqj_a74ef3e6-b26b-470a-abc1-6e80caa59558/istio-proxy/0.log" Oct 30 17:33:36.948593 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:36.948561 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-bndc9_d4fdb214-ef36-4692-b17c-e7296231f8a2/istio-proxy/0.log" Oct 30 17:33:37.771180 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:33:37.771149 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-qghkn_1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63/istio-proxy/0.log" Oct 30 17:33:54.600081 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:33:54.600043 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:34:21.024106 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:21.024074 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-vg6k7_d953cc88-99a1-4389-aec1-f8cfda8e7de5/manager/0.log" Oct 30 17:34:21.429943 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:21.429851 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-vt287_2802841b-377f-409c-ba62-eb90a400265a/manager/0.log" Oct 30 17:34:21.876146 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:21.876117 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-zgr4b_aa757e1d-7dbd-47cf-bc6f-54dacda93285/server/0.log" Oct 30 17:34:22.071001 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:22.070968 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-f87fcf56c-5fmhl_7ac575cb-7c53-489b-b05c-09ca444bfabb/manager/0.log" Oct 30 17:34:22.286116 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:22.286082 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-observability-584f99c788-k7579_02309873-b9e3-4457-85d2-75c94131b212/manager/0.log" Oct 30 17:34:22.975845 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:22.975818 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-xpdtn_5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22/copy-manifests/0.log" Oct 30 17:34:23.017269 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:23.017243 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-cbcd98d6d-xpdtn_5269c1d7-fe1e-40c5-9a17-4f6d2fbe2f22/manager/0.log" Oct 30 17:34:24.597239 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:34:24.597195 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:34:33.773462 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:33.773433 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-595cc648c-4wp2c_0c1d6a5b-d18b-40af-8199-2ddc02dae6dd/maas-api/0.log" Oct 30 17:34:45.353597 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:45.353561 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-rkmxm_c4b56d9a-9f54-417a-89d0-bde1880c0dda/postgres/0.log" Oct 30 17:34:54.597925 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:34:54.597888 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:34:57.135081 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:57.135047 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6xzjs_79795cb8-7a9c-4aa0-8bd1-1dec782af242/kuadrant-console-plugin/0.log" Oct 30 17:34:58.322533 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:58.322502 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-kz9k9_ffda5415-4ff3-4fce-abfc-c8e44f905c5b/istio-proxy/0.log" Oct 30 17:34:59.116704 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:59.116668 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-4nbqj_a74ef3e6-b26b-470a-abc1-6e80caa59558/istio-proxy/0.log" Oct 30 17:34:59.720822 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:34:59.720792 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-bndc9_d4fdb214-ef36-4692-b17c-e7296231f8a2/istio-proxy/0.log" Oct 30 17:35:00.505384 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:35:00.505350 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-qghkn_1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63/istio-proxy/0.log" Oct 30 17:35:24.597931 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:35:24.597845 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:35:54.597141 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:35:54.597098 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:36:24.596872 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:36:24.596830 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:36:54.595566 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:36:54.595530 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:37:24.596213 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:37:24.596176 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:37:54.596346 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:37:54.596254 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:38:11.912504 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:11.912427 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj"] Oct 30 17:38:11.913064 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:11.912645 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" containerID="cri-o://af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576" gracePeriod=30 Oct 30 17:38:11.923890 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:11.923858 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn"] Oct 30 17:38:11.924103 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:11.924081 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" containerID="cri-o://c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e" gracePeriod=30 Oct 30 17:38:17.164483 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.164459 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:38:17.198947 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.198920 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:38:17.238792 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.238765 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-socket\") pod \"a74ef3e6-b26b-470a-abc1-6e80caa59558\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " Oct 30 17:38:17.238956 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.238800 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-data\") pod \"a74ef3e6-b26b-470a-abc1-6e80caa59558\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " Oct 30 17:38:17.238956 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.238868 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a74ef3e6-b26b-470a-abc1-6e80caa59558-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74ef3e6-b26b-470a-abc1-6e80caa59558-istiod-ca-cert\") pod \"a74ef3e6-b26b-470a-abc1-6e80caa59558\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " Oct 30 17:38:17.238956 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.238900 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-kube-api-access-pj2ft\" (UniqueName: \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-kube-api-access-pj2ft\") pod \"a74ef3e6-b26b-470a-abc1-6e80caa59558\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " Oct 30 17:38:17.238956 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.238932 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-credential-socket\") pod \"a74ef3e6-b26b-470a-abc1-6e80caa59558\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " Oct 30 17:38:17.239190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.238987 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-envoy\") pod \"a74ef3e6-b26b-470a-abc1-6e80caa59558\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " Oct 30 17:38:17.239190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239021 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-token\" (UniqueName: \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-token\") pod \"a74ef3e6-b26b-470a-abc1-6e80caa59558\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " Oct 30 17:38:17.239190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239049 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-podinfo\") pod \"a74ef3e6-b26b-470a-abc1-6e80caa59558\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " Oct 30 17:38:17.239190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239068 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-certs\") pod \"a74ef3e6-b26b-470a-abc1-6e80caa59558\" (UID: \"a74ef3e6-b26b-470a-abc1-6e80caa59558\") " Oct 30 17:38:17.239190 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239088 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-socket" pod "a74ef3e6-b26b-470a-abc1-6e80caa59558" (UID: "a74ef3e6-b26b-470a-abc1-6e80caa59558"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.239534 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239258 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a74ef3e6-b26b-470a-abc1-6e80caa59558-istiod-ca-cert" pod "a74ef3e6-b26b-470a-abc1-6e80caa59558" (UID: "a74ef3e6-b26b-470a-abc1-6e80caa59558"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:17.239652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239531 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-certs" pod "a74ef3e6-b26b-470a-abc1-6e80caa59558" (UID: "a74ef3e6-b26b-470a-abc1-6e80caa59558"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.239652 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239546 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-data" pod "a74ef3e6-b26b-470a-abc1-6e80caa59558" (UID: "a74ef3e6-b26b-470a-abc1-6e80caa59558"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.239917 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239773 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-credential-socket" pod "a74ef3e6-b26b-470a-abc1-6e80caa59558" (UID: "a74ef3e6-b26b-470a-abc1-6e80caa59558"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.239917 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239856 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a74ef3e6-b26b-470a-abc1-6e80caa59558-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.239917 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239869 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.239917 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239884 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.239917 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239893 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.239917 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.239901 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.241624 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.241574 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-podinfo" pod "a74ef3e6-b26b-470a-abc1-6e80caa59558" (UID: "a74ef3e6-b26b-470a-abc1-6e80caa59558"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:17.241949 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.241927 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-envoy" pod "a74ef3e6-b26b-470a-abc1-6e80caa59558" (UID: "a74ef3e6-b26b-470a-abc1-6e80caa59558"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.242030 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.241968 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-kube-api-access-pj2ft" pod "a74ef3e6-b26b-470a-abc1-6e80caa59558" (UID: "a74ef3e6-b26b-470a-abc1-6e80caa59558"). InnerVolumeSpecName "kube-api-access-pj2ft". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.242203 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.242180 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-token" pod "a74ef3e6-b26b-470a-abc1-6e80caa59558" (UID: "a74ef3e6-b26b-470a-abc1-6e80caa59558"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.340782 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-certs\") pod \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.340828 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istiod-ca-cert\") pod \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.340866 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-token\" (UniqueName: \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-token\") pod \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.340907 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-kube-api-access-2dfhw\" (UniqueName: \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-kube-api-access-2dfhw\") pod \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341005 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-socket\") pod \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341056 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-data\") pod \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341084 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-credential-socket\") pod \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341130 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-podinfo\") pod \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341163 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-envoy\") pod \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\" (UID: \"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63\") " Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341452 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj2ft\" (UniqueName: \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-kube-api-access-pj2ft\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341474 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341488 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341501 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a74ef3e6-b26b-470a-abc1-6e80caa59558-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341596 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-data" pod "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" (UID: "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341729 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-credential-socket" pod "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" (UID: "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341912 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-certs" pod "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" (UID: "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.341976 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.341930 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istiod-ca-cert" pod "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" (UID: "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:17.343244 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.342728 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-socket" pod "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" (UID: "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.344583 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.344560 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-kube-api-access-2dfhw" pod "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" (UID: "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63"). InnerVolumeSpecName "kube-api-access-2dfhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.344732 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.344706 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-podinfo" pod "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" (UID: "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:17.344945 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.344738 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-token" pod "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" (UID: "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:17.345146 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.345129 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-envoy" pod "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" (UID: "1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:17.442287 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.442259 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.442287 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.442284 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.442287 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.442293 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.442520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.442304 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.442520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.442314 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.442520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.442322 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.442520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.442331 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.442520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.442338 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.442520 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.442346 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dfhw\" (UniqueName: \"kubernetes.io/projected/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63-kube-api-access-2dfhw\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:17.608991 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.608913 6400 generic.go:356] "Generic (PLEG): container finished" podID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerID="af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576" exitCode=0 Oct 30 17:38:17.608991 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.608973 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" Oct 30 17:38:17.609201 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.608997 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" event={"ID":"a74ef3e6-b26b-470a-abc1-6e80caa59558","Type":"ContainerDied","Data":"af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576"} Oct 30 17:38:17.609201 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.609053 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj" event={"ID":"a74ef3e6-b26b-470a-abc1-6e80caa59558","Type":"ContainerDied","Data":"c6ab999741cd514f4bf9286e87103b42842e43abd0126126c457b3f202f15f23"} Oct 30 17:38:17.609201 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.609086 6400 scope.go:122] "RemoveContainer" containerID="af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576" Oct 30 17:38:17.610468 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.610445 6400 generic.go:356] "Generic (PLEG): container finished" podID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerID="c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e" exitCode=0 Oct 30 17:38:17.610585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.610490 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" Oct 30 17:38:17.610585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.610523 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" event={"ID":"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63","Type":"ContainerDied","Data":"c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e"} Oct 30 17:38:17.610585 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.610568 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn" event={"ID":"1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63","Type":"ContainerDied","Data":"16a48ffe16b6c91f461e8ec81427744a9f5cb70307ad9bdd95fd9b19cd62d681"} Oct 30 17:38:17.619041 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.619026 6400 scope.go:122] "RemoveContainer" containerID="af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576" Oct 30 17:38:17.619323 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:38:17.619305 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576\": container with ID starting with af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576 not found: ID does not exist" containerID="af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576" Oct 30 17:38:17.619402 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.619336 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576"} err="failed to get container status \"af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576\": rpc error: code = NotFound desc = could not find container \"af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576\": container with ID starting with af8298dd06627dc44812050b5d04f6a23b26c6715a629d3f3742101205abd576 not found: ID does not exist" Oct 30 17:38:17.619402 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.619357 6400 scope.go:122] "RemoveContainer" containerID="c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e" Oct 30 17:38:17.628418 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.628402 6400 scope.go:122] "RemoveContainer" containerID="c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e" Oct 30 17:38:17.628688 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:38:17.628669 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e\": container with ID starting with c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e not found: ID does not exist" containerID="c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e" Oct 30 17:38:17.628769 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.628695 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e"} err="failed to get container status \"c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e\": rpc error: code = NotFound desc = could not find container \"c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e\": container with ID starting with c8f7a0ea62b112a7fb343a37588e2b460ae5776fbdf4feba82d0909a96640b6e not found: ID does not exist" Oct 30 17:38:17.672798 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.672775 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj"] Oct 30 17:38:17.684455 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.684433 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-4nbqj"] Oct 30 17:38:17.717854 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.717830 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn"] Oct 30 17:38:17.729625 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:17.729589 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-qghkn"] Oct 30 17:38:18.965893 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:18.965856 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" path="/var/lib/kubelet/pods/1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63/volumes" Oct 30 17:38:18.966355 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:18.966337 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" path="/var/lib/kubelet/pods/a74ef3e6-b26b-470a-abc1-6e80caa59558/volumes" Oct 30 17:38:24.596259 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:38:24.596221 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:38:26.959479 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:26.959439 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9"] Oct 30 17:38:26.959952 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:26.959788 6400 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" containerID="cri-o://3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a" gracePeriod=30 Oct 30 17:38:32.210798 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.210773 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:38:32.381672 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.381569 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-data\") pod \"d4fdb214-ef36-4692-b17c-e7296231f8a2\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " Oct 30 17:38:32.381840 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.381690 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-token\" (UniqueName: \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-token\") pod \"d4fdb214-ef36-4692-b17c-e7296231f8a2\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " Oct 30 17:38:32.381840 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.381733 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-podinfo\") pod \"d4fdb214-ef36-4692-b17c-e7296231f8a2\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " Oct 30 17:38:32.381840 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.381764 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-kube-api-access-9p8zr\" (UniqueName: \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-kube-api-access-9p8zr\") pod \"d4fdb214-ef36-4692-b17c-e7296231f8a2\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " Oct 30 17:38:32.381840 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.381790 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-certs\") pod \"d4fdb214-ef36-4692-b17c-e7296231f8a2\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " Oct 30 17:38:32.381840 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.381826 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-socket\") pod \"d4fdb214-ef36-4692-b17c-e7296231f8a2\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " Oct 30 17:38:32.382117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.381854 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-envoy\") pod \"d4fdb214-ef36-4692-b17c-e7296231f8a2\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " Oct 30 17:38:32.382117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.381877 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-data" pod "d4fdb214-ef36-4692-b17c-e7296231f8a2" (UID: "d4fdb214-ef36-4692-b17c-e7296231f8a2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:32.382117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.381894 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-credential-socket\") pod \"d4fdb214-ef36-4692-b17c-e7296231f8a2\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " Oct 30 17:38:32.382117 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.381945 6400 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4fdb214-ef36-4692-b17c-e7296231f8a2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdb214-ef36-4692-b17c-e7296231f8a2-istiod-ca-cert\") pod \"d4fdb214-ef36-4692-b17c-e7296231f8a2\" (UID: \"d4fdb214-ef36-4692-b17c-e7296231f8a2\") " Oct 30 17:38:32.382332 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.382162 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-certs" pod "d4fdb214-ef36-4692-b17c-e7296231f8a2" (UID: "d4fdb214-ef36-4692-b17c-e7296231f8a2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:32.382332 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.382280 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-credential-socket" pod "d4fdb214-ef36-4692-b17c-e7296231f8a2" (UID: "d4fdb214-ef36-4692-b17c-e7296231f8a2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:32.382434 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.382353 6400 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-certs\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:32.382434 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.382373 6400 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-credential-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:32.382434 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.382383 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-socket" pod "d4fdb214-ef36-4692-b17c-e7296231f8a2" (UID: "d4fdb214-ef36-4692-b17c-e7296231f8a2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:32.382434 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.382389 6400 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-data\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:32.382624 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.382470 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4fdb214-ef36-4692-b17c-e7296231f8a2-istiod-ca-cert" pod "d4fdb214-ef36-4692-b17c-e7296231f8a2" (UID: "d4fdb214-ef36-4692-b17c-e7296231f8a2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 17:38:32.384235 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.384204 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-podinfo" pod "d4fdb214-ef36-4692-b17c-e7296231f8a2" (UID: "d4fdb214-ef36-4692-b17c-e7296231f8a2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 17:38:32.384374 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.384251 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-envoy" pod "d4fdb214-ef36-4692-b17c-e7296231f8a2" (UID: "d4fdb214-ef36-4692-b17c-e7296231f8a2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 17:38:32.384647 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.384622 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-token" pod "d4fdb214-ef36-4692-b17c-e7296231f8a2" (UID: "d4fdb214-ef36-4692-b17c-e7296231f8a2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:32.384647 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.384635 6400 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-kube-api-access-9p8zr" pod "d4fdb214-ef36-4692-b17c-e7296231f8a2" (UID: "d4fdb214-ef36-4692-b17c-e7296231f8a2"). InnerVolumeSpecName "kube-api-access-9p8zr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 17:38:32.483420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.483379 6400 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-token\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:32.483420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.483410 6400 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-podinfo\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:32.483420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.483420 6400 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9p8zr\" (UniqueName: \"kubernetes.io/projected/d4fdb214-ef36-4692-b17c-e7296231f8a2-kube-api-access-9p8zr\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:32.483420 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.483429 6400 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-workload-socket\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:32.483737 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.483438 6400 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4fdb214-ef36-4692-b17c-e7296231f8a2-istio-envoy\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:32.483737 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.483447 6400 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4fdb214-ef36-4692-b17c-e7296231f8a2-istiod-ca-cert\") on node \"ip-10-0-128-181.ec2.internal\" DevicePath \"\"" Oct 30 17:38:32.687504 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.687412 6400 generic.go:356] "Generic (PLEG): container finished" podID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerID="3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a" exitCode=0 Oct 30 17:38:32.687504 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.687460 6400 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" Oct 30 17:38:32.687796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.687508 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" event={"ID":"d4fdb214-ef36-4692-b17c-e7296231f8a2","Type":"ContainerDied","Data":"3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a"} Oct 30 17:38:32.687796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.687566 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9" event={"ID":"d4fdb214-ef36-4692-b17c-e7296231f8a2","Type":"ContainerDied","Data":"ae5d7954c9d95bd0f706e731e6119d15481e6ae21c5184339484aeaf3e61680a"} Oct 30 17:38:32.687796 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.687594 6400 scope.go:122] "RemoveContainer" containerID="3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a" Oct 30 17:38:32.697105 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.697088 6400 scope.go:122] "RemoveContainer" containerID="3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a" Oct 30 17:38:32.697373 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:38:32.697353 6400 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a\": container with ID starting with 3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a not found: ID does not exist" containerID="3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a" Oct 30 17:38:32.697430 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.697383 6400 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a"} err="failed to get container status \"3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a\": rpc error: code = NotFound desc = could not find container \"3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a\": container with ID starting with 3e9292c69d73f3386b95ec1b37ae78a532316d2a80314295acbdfaf6d105798a not found: ID does not exist" Oct 30 17:38:32.739428 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.739398 6400 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9"] Oct 30 17:38:32.751237 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.751216 6400 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-bndc9"] Oct 30 17:38:32.966003 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:32.965928 6400 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" path="/var/lib/kubelet/pods/d4fdb214-ef36-4692-b17c-e7296231f8a2/volumes" Oct 30 17:38:48.864584 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:48.864553 6400 ???:1] "http: TLS handshake error from 10.0.128.181:44412: EOF" Oct 30 17:38:48.869255 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:48.869232 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-26nhz_da3c94aa-c880-4844-8cc8-2182ff86e321/global-pull-secret-syncer/0.log" Oct 30 17:38:49.952139 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:49.952111 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-glrfx_12134b5c-de72-4972-a185-3d5450c7b5e7/konnectivity-agent/0.log" Oct 30 17:38:50.217212 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:50.217123 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-128-181.ec2.internal_3c2020069c5384399725929d6ee8319e/haproxy/0.log" Oct 30 17:38:54.598193 ip-10-0-128-181 kubenswrapper[6400]: E1030 17:38:54.598108 6400 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6qzkb" containerName="extractor" Oct 30 17:38:57.096714 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:57.096685 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-6xzjs_79795cb8-7a9c-4aa0-8bd1-1dec782af242/kuadrant-console-plugin/0.log" Oct 30 17:38:58.531042 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.531002 6400 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv"] Oct 30 17:38:58.531456 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.531413 6400 cpu_manager.go:414] "containerMap: removing container" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" Oct 30 17:38:58.531456 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.531428 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" Oct 30 17:38:58.531456 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.531436 6400 cpu_manager.go:414] "containerMap: removing container" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" Oct 30 17:38:58.531456 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.531441 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" Oct 30 17:38:58.531618 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.531470 6400 cpu_manager.go:414] "containerMap: removing container" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" Oct 30 17:38:58.531618 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.531474 6400 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" Oct 30 17:38:58.531618 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.531537 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4fdb214-ef36-4692-b17c-e7296231f8a2" containerName="istio-proxy" Oct 30 17:38:58.531618 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.531548 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0a94a6-f67e-4fa4-9e2e-c1fb2e3cec63" containerName="istio-proxy" Oct 30 17:38:58.531618 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.531554 6400 memory_manager.go:365] "RemoveStaleState removing state" podUID="a74ef3e6-b26b-470a-abc1-6e80caa59558" containerName="istio-proxy" Oct 30 17:38:58.534675 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.534635 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.549393 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.549368 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vstlf\"/\"kube-root-ca.crt\"" Oct 30 17:38:58.549514 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.549404 6400 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-vstlf\"/\"openshift-service-ca.crt\"" Oct 30 17:38:58.561344 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.561321 6400 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-vstlf\"/\"default-dockercfg-mtr8p\"" Oct 30 17:38:58.563083 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.563061 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv"] Oct 30 17:38:58.603975 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.603950 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-proc\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.604094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.603992 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrmr4\" (UniqueName: \"kubernetes.io/projected/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-kube-api-access-jrmr4\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.604094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.604016 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-sys\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.604094 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.604071 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-podres\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.604277 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.604175 6400 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-lib-modules\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.705329 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.705305 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-sys\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.705329 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.705330 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-podres\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.705518 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.705359 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-lib-modules\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.705518 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.705416 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-proc\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.705518 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.705420 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-sys\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.705518 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.705447 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-podres\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.705518 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.705473 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-proc\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.705518 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.705495 6400 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrmr4\" (UniqueName: \"kubernetes.io/projected/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-kube-api-access-jrmr4\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.705777 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.705500 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-lib-modules\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.728557 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.728529 6400 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrmr4\" (UniqueName: \"kubernetes.io/projected/0241dcb3-cca5-4b67-b6f5-7b1e236b5000-kube-api-access-jrmr4\") pod \"perf-node-gather-daemonset-xn8mv\" (UID: \"0241dcb3-cca5-4b67-b6f5-7b1e236b5000\") " pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.847675 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.847587 6400 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:58.887158 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:58.886225 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d90c80ab-29ef-4083-bd11-74e4ba0c297b/alertmanager/0.log" Oct 30 17:38:59.003227 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.003180 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d90c80ab-29ef-4083-bd11-74e4ba0c297b/config-reloader/0.log" Oct 30 17:38:59.121624 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.121525 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d90c80ab-29ef-4083-bd11-74e4ba0c297b/kube-rbac-proxy-web/0.log" Oct 30 17:38:59.147113 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.147088 6400 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv"] Oct 30 17:38:59.152288 ip-10-0-128-181 kubenswrapper[6400]: W1030 17:38:59.152254 6400 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod0241dcb3_cca5_4b67_b6f5_7b1e236b5000.slice/crio-632feb64a9ed25346a8f93dc9ac7b023cac4c049922b33e53dbf57959cf1cdbc WatchSource:0}: Error finding container 632feb64a9ed25346a8f93dc9ac7b023cac4c049922b33e53dbf57959cf1cdbc: Status 404 returned error can't find the container with id 632feb64a9ed25346a8f93dc9ac7b023cac4c049922b33e53dbf57959cf1cdbc Oct 30 17:38:59.241235 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.241214 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d90c80ab-29ef-4083-bd11-74e4ba0c297b/kube-rbac-proxy/0.log" Oct 30 17:38:59.356409 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.356368 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d90c80ab-29ef-4083-bd11-74e4ba0c297b/kube-rbac-proxy-metric/0.log" Oct 30 17:38:59.475384 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.475314 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d90c80ab-29ef-4083-bd11-74e4ba0c297b/prom-label-proxy/0.log" Oct 30 17:38:59.591526 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.591496 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_d90c80ab-29ef-4083-bd11-74e4ba0c297b/init-config-reloader/0.log" Oct 30 17:38:59.821264 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.821230 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" event={"ID":"0241dcb3-cca5-4b67-b6f5-7b1e236b5000","Type":"ContainerStarted","Data":"9122a1fc5c527dc09a142c380c568d1fab0b3363f7e97fceef60f5b81d1469ad"} Oct 30 17:38:59.821472 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.821292 6400 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" event={"ID":"0241dcb3-cca5-4b67-b6f5-7b1e236b5000","Type":"ContainerStarted","Data":"632feb64a9ed25346a8f93dc9ac7b023cac4c049922b33e53dbf57959cf1cdbc"} Oct 30 17:38:59.821472 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.821408 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:38:59.834929 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.834902 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ctcjb_a2b8eb86-b9d8-4e70-8c4e-736070d94081/kube-state-metrics/0.log" Oct 30 17:38:59.864041 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.863998 6400 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" podStartSLOduration=1.8639850949999999 podStartE2EDuration="1.863985095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 17:38:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 17:38:59.862507831 +0000 UTC m=+4649.012788844" watchObservedRunningTime="2026-10-30 17:38:59.863985095 +0000 UTC m=+4649.014266083" Oct 30 17:38:59.950316 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:38:59.950282 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ctcjb_a2b8eb86-b9d8-4e70-8c4e-736070d94081/kube-rbac-proxy-main/0.log" Oct 30 17:39:00.072197 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:00.072125 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-ctcjb_a2b8eb86-b9d8-4e70-8c4e-736070d94081/kube-rbac-proxy-self/0.log" Oct 30 17:39:00.191482 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:00.191449 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-5f87899b87-ckzpf_9f3cf821-229a-4690-abad-332006a3882a/metrics-server/0.log" Oct 30 17:39:00.309702 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:00.309671 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-hlrgr_9b24f224-6dab-4478-890b-9da0d0679187/monitoring-plugin/0.log" Oct 30 17:39:01.142085 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:01.142053 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vgfj7_31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/node-exporter/0.log" Oct 30 17:39:01.258865 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:01.258836 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vgfj7_31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/kube-rbac-proxy/0.log" Oct 30 17:39:01.375863 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:01.375840 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vgfj7_31a0ffa2-1bbf-4d1b-8a41-e0a8b709914a/init-textfile/0.log" Oct 30 17:39:01.496457 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:01.496422 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5sc57_e01c7dab-d74b-4efe-9a0b-58a9eba6d956/kube-rbac-proxy-main/0.log" Oct 30 17:39:01.617729 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:01.617705 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5sc57_e01c7dab-d74b-4efe-9a0b-58a9eba6d956/kube-rbac-proxy-self/0.log" Oct 30 17:39:01.733994 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:01.733946 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-5sc57_e01c7dab-d74b-4efe-9a0b-58a9eba6d956/openshift-state-metrics/0.log" Oct 30 17:39:05.836340 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:05.836309 6400 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-vstlf/perf-node-gather-daemonset-xn8mv" Oct 30 17:39:07.386328 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:07.386301 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-57cc7b5857-4p6bm_a2a21dfa-119a-4ebb-a58c-09b5e3cb3d37/console/0.log" Oct 30 17:39:11.242617 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:11.242567 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qhm42_c8e15255-91bb-411a-9c47-3c714735e70d/dns/0.log" Oct 30 17:39:11.356773 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:11.356736 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qhm42_c8e15255-91bb-411a-9c47-3c714735e70d/kube-rbac-proxy/0.log" Oct 30 17:39:11.728129 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:11.728038 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-68psh_cb25da1d-2305-4f75-9920-e52fe596f5b7/dns-node-resolver/0.log" Oct 30 17:39:11.844397 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:11.844357 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-kz9k9_ffda5415-4ff3-4fce-abfc-c8e44f905c5b/istio-proxy/0.log" Oct 30 17:39:13.315666 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:13.315635 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-rns4g_72976414-f225-493b-9c55-3ab89bb831a8/node-ca/0.log" Oct 30 17:39:15.360121 ip-10-0-128-181 kubenswrapper[6400]: I1030 17:39:15.360087 6400 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-67c76445c-kz9k9_ffda5415-4ff3-4fce-abfc-c8e44f905c5b/istio-proxy/0.log"